Object-oriented machine learning governance

ABSTRACT

Provided is a process including: writing, with a computing system, a first plurality of classes using object-oriented modelling of modelling methods; writing, with the computing system, a second plurality of classes using object-oriented modelling of governance; scanning, with the computing system, a set of libraries collectively containing both modelling object classes among the first plurality of classes and governance classes among the second plurality of classes to determine class definition information; using, with the computing system, at least some of the class definition information to produce object manipulation functions, wherein the object manipulation functions allow a governance system to access methods and attributes of classes among first plurality of classes or the second plurality of classes to manipulate objects of at least some of the modelling object classes; and using at least some of the class definition information to effectuate access to the object manipulation functions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 16/891,863, filed 3 Jun. 2020, titledOBJECT-ORIENTED MACHINE LEARNING GOVERNANCE, which claims the benefit ofU.S. Provisional Patent Application 62/856,713, filed 3 Jun. 2019,titled OBJECT-ORIENTED AI MODELING. The entire content of eachaforementioned application is hereby incorporated by reference.

BACKGROUND 1. Field

The present disclosure relates generally to predictive computer modelsand, more specifically, to the creation and operation of numerousmachine learning (and other types of artificial intelligence) models ina form that is modular, reusable, easier to maintain, faster todevelopment, easily adapted and modified by data scientists and analystsalike.

2. Description of the Related Art

Data scientists today often struggle to manage predictive models thatnumber in the dozens or hundreds. This is the case because of theever-increasing appetite for predictive analytics, fueled by theexistence of open source tools and libraries. Such computer models areoften created or trained on different datasets, by those seeking toprovide support and insights for different objectives. Some companiesrun tens of thousands of propensity models every month just to keep up.

Not only is the development of machine learning models time consuming,it creates a legacy of technology debt, and the constant drive toimprove performance yields to the processing of the datasets isinconsistent from application to applications. The choice of featuremanagement, labeling, selection of positive and negative machinelearning classes, decision on which data is duplicated or redundant,which should be abstracted or aggregated varies from implementation toimplementation.

Existing computer systems for managing machine-learning models aretypically ill-suited for the diversity of contemporary models. In a widerange of use cases, machine learning models are used to predict hownon-deterministic entities or complex entities with emergent propertieswill behave under specific circumstances. Machine learning has taken abespoke and monolithic approach at times, confounding the selection ofan algorithm tweaked to situation at end (a dataset and a businessobjective) as “modelling” rather than creating a functional andadaptable pipeline leveraging subject matter expertise for features,workflows, and ontologies. While the world of software design has movedto microservices, refactoring large applications into independentlydeployable services allowing scaling of releases and more complexintegration, the machine learning world has been left behind, relyingprimarily on notebooks and single purpose/single flow models.

Indeed, prior manual approaches leave much to be desired. Merelyautomating such approaches fails to arm developers with tools needed foremerging levels of complexity in this field. Manual approaches oftenfail to reveal and account for higher-level classifications offunctionality that will be useful for managing complexity inmachine-learning deployments and reasoning about related code and modelsimplemented by that code. As a result, such manual approaches, even ifautomated, often fail to surface modularity in a corpus of models andsufficiently facilitate re-use of and adaptation of models in relateduse cases.

SUMMARY

The following is a non-exhaustive listing of some aspects of the presenttechniques. These and other aspects are described in the followingdisclosure.

Some aspects include a process, including: writing, with a computingsystem, a first plurality of classes using object-oriented modelling ofmodelling methods; writing, with the computing system, a secondplurality of classes using object-oriented modelling of governance;scanning, with the computing system, a set of libraries collectivelycontaining both modelling object classes among the first plurality ofclasses and governance classes among the second plurality of classes todetermine class definition information; using, with the computingsystem, at least some of the class definition information to produceobject manipulation functions, wherein the object manipulation functionsallow a governance system to access methods and attributes of classesamong first plurality of classes or the second plurality of classes tomanipulate objects of at least some of the modelling object classes; andusing at least some of the class definition information to effectuateaccess to the object manipulation functions.

Some aspects include a process, including encoding a quality managementprocess within model objects, where the encoded quality managementprocess expands between data quality to model quality, score(performance), label (ontology governance), or bias.

Some aspects include a process of performing governance managementprocesses embedded with components (modules) of modeling pipelines. Thisprocess may be used in model design, protection of PersonallyIdentifiable Information (PII) audit, modification, certification,development, validation, testing, deployment, upgrade, and retirement.

Some aspects include a process of decoupling the development of modelingtechniques from the characteristics of the datasets by creating theequivalent of patterns at multiple functional locations.

Some aspects include a process of decoupling labeling techniques fromthe labels themselves, expanding the nature and use of labels usingclasses of patterns and design patterns.

Some aspects include the decomposition of the modeling into racks ofoptions of modeling steps and optimizing the selection of said optionsthrough a constrained or unconstrainted set of objectives.

Some aspects include the use of machine learning techniques andoperational research techniques to reduce a search space.

Some aspects include processes to create sequences of actions thatoptimize in the aggregate and in the individual over pipelineperformance.

Some aspects include a process of decoupling using message passing,decomposition of processing of data, and model transformation intoorganized collections of directed source-to-target mappings andpublish/subscribe paradigm.

Some aspects may apply to a variety of use cases. A use case may predictwhether a consumer is likely to make a purchase and determine whether tocause an advertisement to be conveyed to the consumer, e.g., whether tocause some form of message be to be conveyed to the consumer via email,text message, phone call, mailer, or the like, or a discount should beoffered to the consumer. Another use case may predict whether a consumeris likely to submit a claim under a warranty and determine whether thatconsumer is qualified to be offered a warranty or price of the warranty.Another use case may predict whether the consumer is likely to pay offdebt and determine whether the consumer should be offered a loan orcredit card and terms, like interest rate or amount that can beborrowed. Another use case may predict whether a person is likely tobecome ill and determine whether that person should be offered insuranceor terms of the insurance, like deductible or maximum coverage. Anotheruse case may predict whether an industrial process, like an oilrefinery, plastic manufacturing plant, or pharmaceutical manufacturingplant, is likely to operate out of tolerance and determine whetherpreventative maintenance is warranted.

Some aspects include a tangible, non-transitory, machine-readable mediumstoring instructions that when executed by a data processing apparatuscause the data processing apparatus to perform operations including theabove-mentioned process.

Some aspects include a system, including: one or more processors; andmemory storing instructions that when executed by the processors causethe processors to effectuate operations of the above-mentioned process.

Some aspects include an implementation of the above with a distributedgeneral-purpose cluster-computing framework, such as Apache Spark,Apache Storm, Apache Hadoop, Apache Flink, Apache hive, Splunk, amazonKinesis, SQL stream, or Elasticsearch.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned aspects and other aspects of the present techniqueswill be better understood when the present application is read in viewof the following figures in which like numbers indicate similar oridentical elements:

FIG. 1 is a block logical and physical architecture diagram showing anembodiment of a controller in accordance with some of the presenttechniques;

FIG. 2 is a block diagram showing an embodiment of an object-orientationorchestrator in accordance with some of the present techniques;

FIG. 3 is a block diagram showing an embodiment of a modeling pillar inaccordance with some of the present techniques;

FIG. 4 is a block diagram showing an embodiment of a data orchestratorrack in accordance with some of the present techniques;

FIG. 5 is a block diagram showing an embodiment of an orchestrationsystem in accordance with some of the present techniques;

FIG. 6 is a flowchart showing an example of a pillar orchestration inaccordance with some of the present techniques;

FIG. 7 is a block diagram showing an embodiment of an OOM classes inaccordance with some of the present techniques;

FIG. 8 is a flowchart showing an example of the concept ofcontextualization and validation in accordance with some of the presenttechniques;

FIG. 9 is a block diagram showing an embodiment of a model object inaccordance with some of the present techniques;

FIG. 10 is a block diagram showing an embodiment of a modelor collectionused to create an optimized pipeline in accordance with some of thepresent techniques;

FIG. 11 is a block diagram showing an embodiment of an optimized modelorcollection in accordance with some of the present techniques;

FIG. 12 is a block diagram showing an embodiment of creation of anintegrated source to target mappings used in an orchestration inaccordance with some of the present techniques;

FIG. 13 is a flowchart showing an example of a process by which atargeted action is determined using an object-orientation orchestrator;

FIG. 14 is a flowchart showing an example of a process by which atargeted action is determined using an optimization system operating anobject-orientation orchestrator;

FIG. 15 is a flowchart showing an example of a process by which atargeted action is determined using a compiler function operating anobject-orientation orchestrator;

FIG. 16 is a flowchart showing an example of a process by which atargeted action is determined using a quality management systemoperating in an object-orientation orchestrator;

FIG. 17 is a flowchart showing an example of a process by which atargeted action is determined using an object-orientation orchestratorbased on a set of governance attributes; and

FIG. 18 shows an example of a computing device by which theabove-described techniques may be implemented.

While the present techniques are susceptible to various modificationsand alternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Thedrawings may not be to scale. It should be understood, however, that thedrawings and detailed description thereto are not intended to limit thepresent techniques to the particular form disclosed, but to thecontrary, the intention is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the presenttechniques as defined by the appended claims.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

To mitigate the problems described herein, the inventors had to bothinvent solutions and, in some cases just as importantly, recognizeproblems overlooked (or not yet foreseen) by others in the fields ofcomputer science and data science. Indeed, the inventors wish toemphasize the difficulty of recognizing those problems that are nascentand will become much more apparent in the future should trends inindustry continue as the inventors expect. Further, because multipleproblems are addressed, it should be understood that some embodimentsare problem-specific, and not all embodiments address every problem withtraditional systems described herein or provide every benefit describedherein. That said, improvements that solve various permutations of theseproblems are described below.

FIG. 1 is a schematic block diagram of an example of a controller 10,operating within a computing system 100, in which the present techniquesmay be implemented. In some embodiments, the computing system 100 mayinclude an entity log repository 12, which in some cases may includeentity events 14 and entity attributes 16. Entity events may includetargeted actions, non-targeted actions, or both. The computing system100 may further include a targeted action repository 18 and a pluralityof potential targeted actions 20.

In some embodiments, the entity logs may be in the form of datasets. Insome of the embodiments, there may be four types of datasets: trainingdatasets, validation datasets, test (e.g., quality assurance) datasets,and application (or other types of target) datasets. A training datasetmay be a dataset used to fit parameters of a model or to find patternsin the dataset. Training datasets may include pairs of an input vectorand desired output targets. Based on the result of the comparison andthe specific modelor object being used, the parameters of the modellingsteps resulting from the binding of the modelor with the dataset may beadjusted for the sake of tuning or optimization (e.g., adjustingparameters of a model to move the output of an objective function closerto, or all of the way to, a local or global maximum or minimum). Avalidation dataset may be a dataset of examples used to tune theparameters of a model (modelor object binding with the dataset). A testdataset may be a dataset independent of the training dataset thatexhibits the same, or similar, statistical and semantic properties asthe training dataset. An application dataset may be a datasetindependent of the training dataset used to put a model into production.In some embodiments, a dataset may be repeatedly split between trainingdataset and validation dataset for the purpose of cross-validation.Using association, datasets may be linked to create a datasetassociation. Another use of dataset association may be the associationof target data sets segmented based on attributes. In some embodiments,segmentation may be based on time and datasets may be representations ofperiods of a business, such as week 1 data, week 2 data, etc. Throughdataset association, comparison of performance of models and KeyPerformance Indicator (KPI) over time may be facilitated and may be usedto trigger a retraining regime (itself implemented as a polymorphism insome cases). In some embodiments, different training datasets may beused for different modeling steps on a pipeline. In some embodiments,different validation datasets may be used for different modeling stepson a pipeline.

In some embodiments, obtained raw data may encode, or serve as a basisfor generating, entity logs related to multiple different entities.Examples include records each describing a history of events associatedwith an individual respective entity, e.g., with a one-to-one mapping oflogs to entities or with shared logs. In some cases, these events areevents describing exogenous actions that impinge upon the entity, likemessages sent to the entity, news events, holidays, weather events,political events, changes in vendors to an industrial process, changesin set points to an industrial process, and the like. In some cases,these events describe endogenous actions performed by the entity, like apurchase, a warranty claim, an insurance claim, a default, a payment ofa debt, presenting with a health problem, and out-of-toleranceindustrial process metric, process yield, weather phenomenon, and thelike. In some embodiments, the events are labeled with some indicia ofsequence, like an indicium of time, for instance, with date stamps orother types of timestamps. In some embodiments, the event logs arerecords exported from a customer relationship management system, eachrecord pertaining to a different customer, and which may include theresult of various transformations on such records. In some embodiments,entity events may include targeted actions (e.g., a targeted outcome),non-targeted actions, or both. In some embodiments, the actions includethose described as being selected among in U.S. patent application Ser.No. 15/456,059, titled BUSINESS ARTIFICIAL INTELLIGENCE MANAGEMENTENGINE, the contents of which are hereby incorporated by reference.

In some embodiments, entity logs may further include non-eventattributes. The non-event attributes may include attributes of people,like psychometric or demographic attributes, like age, gender,geolocation of residence, geolocation of a workplace, income, number andage of children, whether they are married, and the like. In someembodiments, the non-event attributes may include attributes of adatacenter, for instance, a cooling capacity, an inventory of HVACequipment therein, a volumetric flow rate maximum for fans, and thelike. In some cases, such attributes may include values indicatingtransient responses to stimulus as well.

In some embodiments, a plurality of potential targeted actions 20 mayinclude business objectives or target states of other non-linear, highlycomplex systems, like some forms of industrial process controls.

In some embodiments, the state to which the controller is responsive(e.g., in online use cases for publishers and subscribers) may beingested in a subject-entity event stream 22. In some embodiments, thestream may be a real time stream, for instance, with data being suppliedas it is obtained (e.g., within less than 10 minutes, 10 seconds, 1second, or 500 milliseconds of being obtained) by, or in relation to,subject entities (e.g. subscribers), for instance, in queries sent asthe data is obtained to request a recommended responsive action in viewof the new information.

In some embodiments, a controller 10 may include a class-basedObject-Oriented Modeling (OOM) orchestrator 24 (which in someembodiments may be referred to with the trade name CEREBRI) built aroundthe concept of objects.

In some embodiments, OOM orchestrator 24 be based, in part, on the broadprinciples of Object-Oriented Programming (OOP). From the perspective ofOOP, there may be differences arising from the inclusion and use ofdatasets, adaptation for machine learning development usage lifecycles,pipelines, self-improvement of models, design through composition, andmultiple-purpose labeling (MUPL). In OOM, the application of the “code”to “data” may, at least in part, modify the code. That may not be thecase in OOP. Unlike OOP, the code structure for OOM may be in multipleprogramming languages.

In some embodiments, OOM orchestrator 24 may implement functionalitythat includes one or more of: abstraction, aggregation, arbitrator,association, accessor, optimization, auditor, binding, orchestration,composition, composition sheets, composition association, ConcurrentOntology Labelling Datastore (COLD), contextualization,cross-contextualization, dataset, dataset association, data streams,encapsulation, governance, inheritance, labelling, messaging, modelor,orchestration, policing, policors, object-oriented modeling (OMM),object-oriented quality management (OQM), object-publish-subscribemodeling (OPSM), pipelining, realization, targeting, and winnowing.

In some embodiments, OOM orchestrator 24 may have various types ofclasses, including: pillars, ontologies, modelor, models, datasets,labels, windows, and customer journeys. In some embodiments, thesetechniques may be implemented in conjunction with the predictive systemsdescribed in U.S. patent application Ser. Nos. 15/456,059; 16/151,136;62/740,858 and 16/127,933, the contents of which are hereby incorporatedby reference, e.g., by leveraging the data models therein, providingoutputs that serve as features thereof, or taking inputs from thesesystems to form input features of the techniques described herein.

In some embodiments related to OOM, a class may be a program-code orprogram-data template used to create objects.

One of the challenges in bringing machine learning and object-orientedconcept together may be managing vocabulary. In machine learning,features may be individual measurable properties or characteristic of aphenomenon being observed. Features may be at times referred to asattributes. In some embodiments, a feature may be referred as anindependent variable, a predictor variable, a covariate, a regressor, ora control variable.

In object-oriented design, attributes may be elements of an object orclass definition. In some embodiments, when a concept may have dualmeaning (ML for machine learning, OO for object Oriented), a prefix ofML or OO may be applied, accordingly. This convention applies, amongothers, to labels, classes, attributes, and methods whenever there isambiguity.

In some embodiments, an element of CEREBRI may be a rack class. A rackis a framework or a pattern for a canonic modeling step without theinstantiated of a data set and application to the dataset.

In some embodiments, an element of CEREBRI may be a modelor class. Amodelor is a framework or a pattern for a modeling step without theinstantiated of a data set and application to the dataset. It may bedecoupled from the modeling step itself. A modeling step may be createdby binding a dataset or a dataset association with a modelor. A modelingstep may be used to perform computation, transformation, or mapping on awhole or a part of a dataset. Modelors that aim to achieve the samefunctionality in a modeling pipeline may belong to the same rack.Modelors may be parametrized. In some embodiments, these parameters maybe attributes. In some embodiments, an attribute set of modelors may begovernance attributes that may be leveraged prospectively orretrospectively. An attribute may be a type of intellectual propertyrights of the modelor code, for example, the type of license of the code(e.g., open source or proprietary).

In some embodiments, an element of CEREBRI may be an orchestration. ACEREBRI orchestration may have two sub-domains: A data domain thattransforms data into objects, and an AI (artificial intelligence) domainthat transforms these objects into results. These results can be score,indexes, lists, ranked order lists among others. Each sub-domain in anOOM based machine learning solution requires a nominal set of steps tobe executed in sequence and/or in parallel.

In some embodiments, orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system. Orchestrations may operate on objects to create newobjects and thus may be recursive. In some embodiments, there are twotypes of orchestration in CEREBRI: modelor orchestrations and modelorchestrations.

In some embodiments, modelor orchestrations may perform manipulationsand computations based on modelor objects to create pipelines. Anorchestration bound to at least one dataset is an example of what isreferred to as a pipeline.

In some embodiments, an element of CEREBRI may be optimization. TheCEREBRI optimization may be akin to a compiler function that selectswhich modelors to be used and what parameters of these modelors shouldemployed. In some embodiments, one or more modelors may be chosen fromeach rack. In some embodiments, no modelor is chosen from at least someof the racks. The parameters of the modelors and the order of the racksmay be optimized for one or more objective functions based on theconstraints of a modelor governance. The time horizon, location horizon,and datasets horizon may be set. The organization of the modelors to beoptimized may be carried out through heuristics, operation research, ormachine learning itself.

In some embodiments, optimization may use one or more of a randomforest-based approach (e.g., an ensemble of models trained withclassification and regression trees (CART)), simulated annealing,reinforcement learning, genetic algorithm, gradient descent, andR-Learning. The optimization methods may include: Bayesian search,Boruta, and Lasso. In some embodiments, optimization methods may beconstrained through hard limitations, governance limitations, andregularization.

In some embodiments, a pipeline may be implemented in at least oneprogramming language. The underlying method used for the computations ormanipulations is an example of what is referred as a sheet. In someembodiments, sheets may be designed through a procedural programmingframework, a hyper-parameter optimization, database queries, scripting,stored procedure, Directed Asynchronous Graph (DAG) ApplicationProgramming Interface (API) calls, or Source to Target Mapping (STM). Insome embodiments, a sheet may be compiled into a single archive allowingruntime to efficiently deploy an entire pipeline (or a selectedportion,), including its classes and their associated resources, in asingle request. In some embodiments, a modeling orchestration may beexecuted using an engine such as stream set, NiFi, pulsar, Kafka, Kafka,RabbitMQ, NATS, Firebase, Pusher, SignalR, Databricks, Socket.IO,OSlsoft Pi, or Heron.

In some embodiments, objects or data being orchestrated through sheetsmay be written in Scala. In some embodiments, objects or data beingorchestrated through sheets may be written in Java. The objects or thedata being orchestrated may be compiled into jar files and stored on aJava Virtual Machine (JVM). In some embodiments, a sheet (configurableand configured by data scientists) of a composition of the objects ordata may be stored in JSON (JavaScript object notation). In someembodiments, a pipeline may be written in Python, Scala, and SQL(structured query language). A sheet may describe the flow of data ormessages from origin objects to destination objects and may also definehow to access, mutate, map, validate, and bind the data, dataset, ormessages along the way. Embodiments may use a single source object torepresent the origin object, multiple processors to mutate data andobjects, and multiple destination stages to represent destinationobject. In some embodiments, embodiments may use an object that triggersa task when it receives a message. To process large volumes of data,embodiments may use multithreaded sheets or cluster-mode sheets.

In some embodiments, when a sheet is generated though a design processusing OOM, embodiments may create a new corresponding sheet for a targetplatform through inheritance.

In some embodiments, a sheet may be compiled into a single archiveallowing runtime to efficiently deploy an entire application, includingits classes and their associated resources, in a single request. In someembodiments, an orchestration may be executed with the aid of aninterpreter.

In some embodiments, a class of modelors may be an example of what isreferred to as pillars. Pillar classes may purport to support elementsof machine learning systems. Pillar classes may answer questions suchas:

-   -   a. Who will engage in an action,    -   b. Commitment or loyalty to a brand,    -   c. Commitment to spend,    -   d. Commitment to tenure (and counter churn away),    -   e. Timing propensity of engagement,    -   f. Affinity for choices of engagement (including location and        channel) from selections,    -   g. Affinity for choices of engagement for synthesized        selections,    -   h. Action to be set engagement,    -   i. Clustering of attributes or behaviors,    -   j. Classification, and    -   k. Product or service recommendations.

In some embodiments, pillars may use advanced modeling, operationresearch, optimization, statistical analysis, and data sciencetechniques (e.g., machine learning modeling techniques MLMTs) that maybe applied to datasets that have been processed through pipelines.Datasets may change throughout pipelines that may include growing andshrinking in sizes, and growing and shrinking in dimensionality.

In some embodiments, different MLMTs may be used, including: OrdinaryLeast Squares Regression (OLSR), Linear Regression, Logistic Regression,Stepwise Regression, Multivariate Adaptive Regression Splines (MARS),Locally Estimated Scatterplot Smoothing (LOESS), Instance-basedAlgorithms, k-Nearest Neighbor (KNN), Learning Vector Quantization(LVQ), Self-Organizing Map (SOM), Locally Weighted Learning (LWL),Regularization Algorithms, Ridge Regression, Least Absolute Shrinkageand Selection Operator (LASSO), Elastic Net, Least-Angle Regression(LARS), Decision Tree Algorithms, Classification and Regression Tree(CART), Iterative Dichotomizer 3 (ID3), C4.5 and C5.0 (differentversions of a powerful approach), Chi-squared Automatic InteractionDetection (CHAID), Decision Stump, M5, Conditional Decision Trees, NaiveBayes, Gaussian Naive Bayes, Multinomial Naive Bayes, AveragedOne-Dependence Estimators (AODE), Bayesian Belief Network (BBN),Bayesian Network (BN), k-Means, k-Medians, Expectation Maximization(EM), Hierarchical Clustering, Association Rule Learning Algorithms,A-priori algorithm, Eclat algorithm, Artificial Neural NetworkAlgorithms, Perceptron, Back-Propagation, Hopfield Network, Radial BasisFunction Network (RBFN), Deep Learning Algorithms, ReinforcementLearning (RL), Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN),Convolutional Neural Network (CNN), Stacked Auto-Encoders,Dimensionality Reduction Algorithms, Principal Component Analysis (PCA),Principal Component Regression (PCR), Partial Least Squares Regression(PLSR), Multidimensional Scaling (MDS), Projection Pursuit, LinearDiscriminant Analysis (LDA), Mixture Discriminant Analysis (MDA),Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis(FDA), Ensemble Algorithms, Boosting, Bootstrapped Aggregation(Bagging), AdaBoost, Stacked Generalization (blending), GradientBoosting Machines (GBM), Gradient Boosted Regression Trees (GBRT),Random Forest, Computational intelligence such as but not limited toevolutionary algorithms, PageRank based methods, Computer Vision (CV),Natural Language Processing (NLP), and Recommender Systems.

In some embodiments, an engagement in an action may be measured throughmonetary propensity techniques, such as the ones described in U.S.patent application Ser. No. 16/127,933, the contents of which are herebyincorporated by reference. The timing or distribution of timing in anaction may be measured through timing propensity techniques such as theones described U.S. Patent Application 62/847,274, the contents of whichare hereby incorporated by reference. The location affinity in an actionmay be measured through journey propensity techniques, such as the onesdescribed in U.S. Patent Application 62/844,338, the contents of whichare hereby incorporated by reference.

In some embodiments, OOM may improve model development, readability,data engineering transformations, applicability to multiple datasets,applicability to multiple contexts, reusability by reducing thedimensionality, and complexity of machine learning program or set ofprograms efficiently. OOM concepts may allow creation of specificinteractions between modelor objects that may create new modelor objectsfor scoring, actioning, listing, or recommending based on data collectedover a collection of items, period of time, geographic intent, orcategorical definitions. Datasets may be bound to modelors to createmodeling steps. Modeling steps may be applied to target datasets torealize predictions, support decisions, find structure in data, findpatterns, detect outliers, and classify items, cluster items, andoptimized objective functions, while maintaining explicit or implicitrestrictions, without always being explicitly programmed to perform thepurposed realizations.

In some embodiments. CEREBRI may use abstract classes. In someembodiments, an abstract class is a superclass (e.g., a parent class)that cannot be instantiated. Embodiments may need to instantiate one ofits child classes to create a new object. Abstract classes may have bothabstract and concrete objects. Abstract objects may contain only objectsignature, while concrete objects may declare a modelor object body aswell, in some embodiments.

In some embodiments, labelling includes adding tags to data. Augmentingdata with labels may make it more informative and more manageable. Oneuse of labelling in object-oriented modeling may be management oflabels, not as elements in a list or table, but as objects allowing acollection of tags to be used. As objects, OO-labels may be managed andorganized through enforced set of grammar and semantic rules. Oneattribute of an OO-label may be a user facing text for sake of userexperience (UX). In some embodiments, OO-labels may be used in a singleontology used for multiple customers, each with a ML-label for theirbusiness realizing a MUPL. OO-labels may encode meta-data about adataset, control information, and governance information among otherthings.

Using a single ontology for multiple customers in CEREBRI may haveadvantages, including the ability to have different representations ofinformation for different purposes. In some embodiments, this techniquemay be used to provide label text in different languages. That said,embodiments are not limited to systems that afford these advantages,which is not to suggest that any other described attributes arelimiting.

In some embodiments, an element of CEREBRI is the Concurrent OntologyLabelling Datastore (COLD) methodology implemented in code of CEREBRI.In CEREBRI OOM, OO-labels may be governed by an ontology (anorganization of information). Ontology in COLD may be domain specific(e.g., in telco or insurance) or domain independent (e.g., in marketingor interactive voice response (IVR)) COLD framework forsuperclass/subclass inheritance. In some embodiments, this approach maybe enforced through a labelling format to a specific grammar or syntaxinside OO-labels via attributes. In some embodiments, OO-labelling maymap a customer/consumer/user facing labels (the ML-label) to a genericontology through a feature of the OO-labels. COLD, in some cases, isexpected to facilitate efforts by data scientists and softwaredevelopers to have a “dual-ledger” view of labels, one internal to thepipeline sheet (that is implementation and coding) and one external forgovernance, quality, or user interface requirements.

In some of the embodiments related to machine learning, new features(which may increase dimension of datasets) may also be obtained from oldfeatures using a method known as feature engineering using knowledgeabout a problem to create new variables. In some embodiments, whilefeature engineering may be part of the application of machine learning,it may be difficult and expensive in terms of time, resource, and talentperspective. In some embodiments, CEREBRI may useobject-publish-subscribe modeling to convey improvements in featureengineering from one object to another.

In some embodiments related to machine learning, features that may notbe helpful in the performance of the models may be removed (which maydecrease in dimensionality of datasets); this is an example of what isreferred to as feature selection. Feature selection may be helpful interms of the required time, resource, and talent perspective. In someembodiments, CEREBRI may use object-publish-subscribe modeling to conveyimprovements in feature selection from one object to another.

In some embodiments, OOM may create a generic feature engineeringmethodology, through polymorphism or inheritance, to optimize on a basisof a business domain or a vertical market.

In some embodiments, polymorphism (i.e., one name in many forms) mayfacilitate reuse of objects (e.g., objects with the same name, or symbolby which they are represented) in use cases implicating differentfunctionality. In some embodiments, polymorphism may be used to declareseveral modelor objects with the same name until the objects aredifferent in certain characteristics, such as context or segment withina dataset. In some embodiments, polymorphism may be used to declareseveral model objects with the same name until the objects are differentin certain characteristics, such as context. By using a modeloroverriding feature, some embodiments may override modelor objects of aparent class from its child class.

In some embodiments, inheritance may facilitate extension of a classwith child classes that inherit attributes and methods of the parentclass. Inheritance may facilitate achieving modeling reusability. Achild class may override values and modelor objects of the parent class.A child class may also add new data and functionality to its parent orshield details of the parent class. Parent classes are also known assuper classes or base classes, while child classes are also known assubclasses or derived classes. Inheritance may extend components withoutany knowledge of the way in which a class was initially implemented, insome embodiments. Declaration options, such as Public and Private, maydictate which members of a superclass may be inherited, in someembodiments. Objects may relate to each other with either a “has a”,“uses a” or an “is a” relationship, wherein the later may be aninheritance relationship, in some embodiments.

In some embodiments, consistency of implementation for datasets may beachieved by using the same polymorphic structures on multiple modelobjects.

In some embodiments, targeting may refer to design or use of a modeloror an orchestration towards a specific goal, purpose, or intent.Targeting of a model may include the definition and availability of atarget dataset.

In some embodiments, datasets may be objects that represent collectionof related set of information composed of individual variables that maybe manipulated individually or collectively. The data inside a datasetmay include sets or collection of examples that may be factors, events,items, or journeys. A journey is a collection of events organized alongone or more timelines.

In some embodiments, an element of CEREBRI may beobject-publish-subscribe modeling (OPSM) framework. In some of theembodiments using OPSM, senders of messages (e.g., publisher objects) donot choose the messages to be sent directly to specific receivers (e.g.,subscriber objects). Publishers may categorize published messages intotopics (which may be objects themselves) without the knowledge of whichsubscriber objects, if any, there are. Messages may be generated basedon logic inside publisher objects. Similarly, subscriber object mayexpress interest in one or more classes and only receive messages thatare of interest, without the knowledge of which publishers, if any,there are. OPSM may be used for feature engineering. OPSM may also beused for introducing new data sources. In some embodiments, OPSM may beused to create an audit trail of performance by having OQM be thepublisher. OPSM may be used to leverage Source-to-Target Mapping (STM).STMs may include sets of data transformation instructions that determinehow to convert the structure and content of elements of datasets in thesource system to the structure and content needed in the target system.STMs may assist modelors in efforts to identify columns or keys in thesource system and point them to columns or keys in the target systems.Additionally, modelors may map data values in the source system to therange of values in the target system. OPSM may allow the processing ofdata in a batch mode, in a streaming mode, or in a combination thereof,in some embodiments.

Topics may be organized through ontologies allowing constructions ofsubtopics, in some embodiments.

In some embodiments, an element of CEREBRI is data-stream. Data streamsmay be a time-sequenced (e.g., time-stamped) set of messages that sharethe same source and the same destination. Data stream may beinstantiated by binding an element of a dataset with another element ofa dataset for the purpose of staging transformations, in someembodiments.

The topic in an OPSM may be a data stream.

In some embodiments, attributes may be in the form of properties,features, contexts, state-machine state, or components among others.

In some embodiments, an auditor may be a specific class that captureshistorical information about multiple aspects of an operation. In someembodiments, auditor may subscribe to the attribute history of keyobjects. Auditors may be used for governance management.

In some embodiments, quality Management (QM) in an object-orientedmodeling paradigm that may be implemented as a process that integratesraw data ingestion, manipulation, transformation, composition, andstorage for building artificial intelligence models. In legacy designs,quality management may include (and in some cases may be limited to)Extract, Transform, and Load (ETL) phases of effort and to the reportingof model performance (e.g., recall, precision, F1, etc.) from an end toend perspective as a quality.

Deposition of design process and operation of models, developed usingOOM into objects, may facilitate efforts to cause quality to be embeddedin objects. Quality may be attributes of objects. Modelor, boundmodelors, and pipelines may be managed through multiple lifecyclesrather than a single one. In some embodiments, Object-Oriented QM (OQM)may have six components:

-   -   a. Data quality monitoring (DQM): DQM measures, not necessarily        exclusively (which is not to suggest that other lists are        exclusive), new or missing data source (table) or data element,        counts, mull count and unique counts, and datatype changes. DQM        may be used to figure out which data sources are reliable.    -   b. Model quality monitoring (MQM): MQM may measure, not        necessarily exclusively (which is not to suggest that other        lists are exclusive), model-based metrics, such as F1,        precision, recall, etc., or data, and triggers retraining for        drift.    -   c. Score quality monitoring (SQM): SQM may perform model        hypothesis tests, including Welch's t-test (e.g., parametric        test for equal means) and the Mann-Whitney U-test (e.g.,        non-parametric test for equal distributions). SQM may also        compute lift tables, a decile table based on the predicted        probability of positive class membership, with the cumulative        distribution function of positive cases added in. The gain chart        is a plot of the cumulative distribution function of positive        cases may be included as a function of quantile.    -   d. Label quality monitoring (LQM): Labels may be categorical and        bound by semantic rules or ontologies. LQM may be used to        understand which data sources are leverageable and impactful.        LQM may be used for data debt management and enhancing        compositions for performance.    -   e. Bias quality monitoring (BQM): Bias is a systematic        distortion of the relationship between a variable, a data set,        and results. Three types of bias may be distinguished:        information bias, selection bias, and confounding, in some        embodiments.    -   f. Private quality monitoring (PQM): Privacy may cover        personally identifiable information and access of privileged        information.

In some embodiments, OQM attributes may include count, unique count,null count, mean, max, min, standard-deviation, median, missing datasource, new data source, missing data element, new data element,sparsity of reward, data type change, Accuracy, Precision, Recall, F1,ROC AUC, TPR, TNR, 1-FPR, 1-FNR, brier gain, 1-KS, lift statistic, CVArea under Curve, 1-CV turn on, CV plateau, 1-brier turn on, brierplateau, MAPk, TkCA, Coverage, entropy coverage, MAPk cohort, TkCApopulation, action percentage change, no action percentage change,action frequency rate, action recency rate, normalized action frequencyrate, normalized action recency rate, expected reward, direct method,inverse propensity method, doubly robust method, weighted doubly robust,sequential doubly robust, magic doubly robust, incremental responserate, net incremental revenue, Mann Whitney Wilcoxon's U test, decileanalysis test, effect size test, and economic efficiency.

In some embodiments, abstraction and encapsulation may be tied together.Abstraction is the development of classes, objects, types in terms oftheir interfaces and functionality, instead of their implementationdetails. Abstraction applies to a modelor, a modeling orchestration, apipeline, datasets, or some other focused representation of items.Encapsulation hides the details of implementation.

In some embodiments, modeling abstraction in CEREBRI may concealcomplexity from users and show them only the relevant information.Modeling abstraction may be used to manage complexity. Data scientistand operation research developers may use abstraction to decomposecomplex systems into smaller components. In some embodiments, if onewants to leverage a propensity pillar modelor object, one may not needto know about its internal working and may just need to know the valueand context for usage. In some embodiments, if a modelor object wants tofactor in a sentiment modeling, embodiments may not need to know aboutits internal natural language processing working, and may just need toknow the value and context for usage in other modelor objects. The samemight be true of CEREBRI classes. Embodiments may hide internalimplementation details by using abstract classes or interfaces. On theabstract level, embodiments may only need to define the modelor objectsignatures (name, performance list, combination restriction,privacy-rules, and parameter list) and let each class implement themindependently.

In some embodiments, encapsulation in CEREBRI may protect the data andconfiguration stored in a class from system-wide access. Encapsulationmay safeguard the internal contents of a class like a real-life capsule.CEREBRI pillars may be implemented as examples of fully encapsulatedclasses. Encapsulation is the hiding of data and of methodsimplementation by restricting access to specific objects. Embodimentsmay implement encapsulation in CEREBRI by keeping the object attributesprivate and providing public access to accessors, mutators, validators,bindors, contextors, and policors to each attribute.

In some embodiments, accessors may be CEREBRI public objects used to askan object about itself. Accessor objects may be not restricted toproperties and may be any public modelor object that gives informationabout the state of the object. When an object is a model or modelor, itmay embed a machine learning state that is ML-State. The state of theobject may be to a large degree a superset the machine learning state ofa model that captures latent, Markov, reward, quality, or governanceinformation.

In some embodiments, mutators may be CEREBRI public objects used tomodify the state of an object, while hiding the implementation ofexactly how the modifications take place. Mutators may be suited forfeature engineering and for source to target mapping.

In some embodiments, contextors may be CEREBRI public objects used tomodify metadata, control, configuration, and data of a dataset object,while hiding the implementation of how the data gets modified. Acontextor may be used to reduce the range of event timing to considerdefining a positive ML-class. Contextors may be used tocross-contextualize datasets used in composition of model objects.

In some embodiments, bindors may be CEREBRI public objects used toassociate a specific dataset to a modelor. Bindors may be a specifictype of mutator. Binding may include the association of modelor object(developed as a single modelor object or the result of a composition) toa dataset or data-stream.

In some embodiments, arbitrators may be CEREBRI public objects used toreplace one object with another one from the same class. Arbitrators maybe used for automatic update of modelor, orchestrations, and pipelines.

In some embodiments, validators may be CEREBRI public objects ensuring,integrity, governance, and quality objects (e.g., quality managementobjects) used to ensure no data linkage or inconsistency of datasets,ML-labels, performance (OQM) and windows of processing. Validators maycheck database consistency applied to aspects of an OOM. A function ofvalidators may be triggering retraining of part or complete pipelinesbased on quality or operational triggers.

In some embodiments, governance in CEREBRI may be a set of structures,processes and policies by which pipeline development, deployment, anduse function within an organization or set of organizations is directed,managed, and controlled to yield business value and to mitigate risk.

In some embodiments, a policy in CEREBRI may refer to set of rules,controls, and resolutions put in place to dictate model behaviorindividually as a whole. Policies are a way governance is encoded andmanaged in some embodiments. The policy items are referred to aspolicors and implemented as CEREBRI objects. As the number of rulesincrease, a policy-driven OOM may suffer from inconsistencies incurredby contradicting rules governing its behavior. In some embodiments,meta-policies (e.g., detection rules) may be used for the detection ofconflicts. In some embodiments, policy relationships may be used. Insome embodiments, attribute-based applicability spaces may be used.

In some embodiments, contextualization in CEREBRI may refer torestriction of datasets to certain ranges in time, space, domain, objecttypes count, users, to accommodate the business and quality requirementof specific use cases. The contextualization may be affecting thedataset used to train (generate) the modelling steps (the one bound witha modelor). Contextualization may be affecting the dataset to which amodel is being applied.

In some embodiments, cross-contextualization in CEREBRI may beimplemented as a process to verify that the datasets used in models arequantitatively and qualitatively compatible and valid. A validity checkmay ensure (or reduce) no data linkage. A dataset used for training maynot be validated on itself or part of itself, in some embodiments. Adataset training may include information available at the moment of thetraining, not future data. Cross-contextualization may be performedthrough contextors, objects that compare datasets across differentpipelines and across different modeling steps.

Winnowing is the concept of limiting the scope or dimensionality of adataset. In some embodiments, winnowing may be achieved through ajudicious use of accessors and mutators. Some embodiments of winnowingmay be in the time domain (e.g., shortening a time range). Someembodiments of winnowing may be geography (e.g., reducing the geographicrange). Some embodiments of winnowing may be ontological (e.g., reducebranches and leaves of a taxonomy, reduce predicates insubject—predicate—object). Some embodiments of winnowing may be binningnumerical attributes into categorical attributes.

In some of embodiments, a publisher objects' subscription may be to aclass of objects or public attributes to the class. This capability ofOOM may be used for feature engineering. In some embodiments, the pillardealing with time optimization (e.g., Té) may be emphasized. Many KPIshave a timing element assigned to them (e.g., propensity to buy have atime dimension assigned to it). Churn may have an inherent timingdimension. Rather that optimizing separately, models for those KPIs maysubscribe to the Té engineered features. If the performance change,relearning will be trigged through validators. Another use of the OPSMis ensembling.

In some embodiments of inheritance, embodiments may implement amulti-channel or omnichannel marketing campaign. The campaign mayleverage to email, mails, in-store displays, or text message. At somelevel, all of these items may be treated the same: All four types mayinvolve creative, cost money to produce, have the same geographicalmarket area and lifetime. However, even though the types may be viewedas the same, they are not identical. An email has email address, a storedisplay does not. Each of these marketing campaign's assets should berepresented by its own class definition. Without inheritance though,each class must independently implement the characteristics that arecommon to channel assets. All assets may be either operational, ready todeploy, or deprecated. Rather than duplicate functionality, inheritanceis expected to facilitate re-use of functionality from another class.

Inheritance may be used to take a modeling step or pipeline and applyingit to a subset of the original dataset. In some embodiments, this isaccomplished by binding to a more restrictive dataset.

Inheritance may be used for feature engineering. In some embodiments,this is accomplished by defining broad features on the superclass andnarrower features in the child class.

In some embodiments, polymorphism may be used to repurpose a model inOOM. In some embodiments, an upsell model (e.g., pushing an existingcustomer to buy a more expensive version of an item she/he owns) may bedeveloped based on a modelor and dataset (one that is bound). Themodel's purpose may be for a business broad customer base. There may be,however, segments of customers within this base. They may include (1)customers who buy on a regular basis, (2) customers who are at risk, and(3) customers who own a specific item. At first glance, these customersmay be treated the same after all. They may have a name, account number,contact information, customer journey. All four types represent rightfultargets for the upsell activities. However, even though the three typesof customers may be viewed as the same, they are not identical becauseof the journeys. For maximum performance, each of these customersegments may be represented by its own class definition.

In some embodiments, CEREBRI may provide two ways to implementpolymorphism: object overloading (e.g., build-time polymorphism) andobject overriding (e.g., run-time polymorphism or dynamic polymorphism).Modelor object overloading happens when various modelor objects with thesame name are present in a class. When they are called, they may bedifferentiated by the number, order, context, and types of parameters.In some embodiments, the type of parameter in the object signature isindication of engineering feature. Modelor object overriding may occurwhen the child class overrides a modelor object of its parent.

Association may be implemented with the act of establishing arelationship between two unrelated classes. A specific type ofassociation is binding. Embodiments may perform association whendeclaring two fields of different types (e.g. car purchase and carservice) within the same class and making them interact with each other.The association may be a one-to-one, one-to-many, many-to-one, ormany-to-many relationship.

One of the associations is dataset association where embodiments mayestablish that multiple datasets are related and binding them to thesame sets of modelors.

Another example association is the association of a development pipelinewith a production pipeline. This may be used to accelerate thetranslation of a pipeline using one coding language for its sheet (e.g.Python) to another (e.g. Scala). This may be used to move from aninterpreted sheet to a compiled sheet.

In some embodiments, aggregation may be a kind of association. It mayoccur when there is a one-way “has a” relationship between the twoclasses associated through their objects. For example, every marketingmessage has a channel (email, mail, or text) but a channel does notnecessarily have a marketing message.

In some embodiments, OOM may cause resulting model objects to managetheir life-cycle autonomously by leveraging dataset association ormodelor association. This ability coupled with micro-services may helpwith operational resiliency.

In some of the embodiments, related to OOM, training may not be the samefor all elements of a pipeline.

In some embodiments, OOM may include the following:

-   -   a. A modeling object KOT performs feature engineering for models        dealing with reducing Churn,    -   b. A modeling object YOR performs KPI estimations for churn        based on model YR and the bound training dataset YMT and the        bound validation dataset YVT,    -   c. The modeling object YOR subscribes to the        time-feature-engineer topic for Churn,    -   d. A Governance object GOT subscribes to all topics,    -   e. The modeling objects KOT find that changing the training set        from YMT to YMT′ (that uses features that at least some of them        are different from YMT features) improves the performance of        models,    -   f. KOT uses OPSM to publish YMT′ as better training model for        churn models,    -   g. Governance object GOT uses polymorphism to create modelor YR′        from YR,    -   h. Modelor YR binds with YMT′ to create YOR′, and    -   i. GOT sends appropriate arbitrator to replace YOR with YOR′.

Another aspect of OOM is the ability of OOM modeling steps or entirepipelines objects to improve their performance semi-autonomously byleveraging association and OOPS. This may be accomplished through OQMassessing the impact of including a new set of data sources, and orfeature engineered attributes.

In some embodiments, OOM may include the following:

-   -   a. A modeling object SOT is performing feature engineering on        the time dimension. Feature engineering may include recency        feature engineering, frequency feature engineering, lag feature        engineering, difference feature engineering, and harmonic        analysis feature engineering, as described in U.S. Patent        Application 62/748,287, the contents of which are incorporated        by reference. Recency features may leverage the last time an        event took place. Lag feature engineering may be used        extensively by organizing timelines into periods. In difference        features engineering, the features are generated by creating the        difference of a given feature of any kind between two consequent        periods or subsequent periods,    -   b. Object SOT uses OPSM to publishes engineered features        definitions as a time-feature-engineer topic,    -   c. Modeling object TOR performs KPI estimations based on modelor        TR bound with training dataset ZMT and validation dataset ZMV,    -   d. Modeling object SOT creates new set of engineered features        FF′,    -   e. Modelor TR spawns modelor TR,    -   f. Modeling object TOR uses inheritance on ZMT to spawn dataset        ZMT′ that includes features FF′.    -   g. Modeling object TOR uses inheritance on ZMV to spawn dataset        ZMV′ that includes features FF′,    -   h. Modeling object TOR uses inheritance to modeling object TOR′,    -   i. Model object TOR binds modelor TOR′ with ZMT′, ZMV′, ZMA′ to        create modeling TR′,    -   j. Quality object QOR uses an accessor to gather the performance        of SQM for TR′ on validation data,    -   k. Quality object QOR compares performance (which can be a        variety of measurements) of SQM for TR′ and TR,    -   l. If the performance of TR′ is worse than performance of TR,        QOR messages to appropriate arbitrator to delete TR′, TOR′,        ZMT′, ZMV′, and    -   m. If the performance of TR′ is better than performance of TR,        QOR messages to appropriate arbitrator to replace ZOR with ZOR′,        ZR with ZR′, ZMT with ZMT′, ZMV with ZMV′.

FIG. 2 is a diagram that illustrates an exemplary architecture 2000 ofobject-oriented orchestration in accordance with some of the embodimentsof the present disclosure. Various portions of systems and methodsdescribed herein may include or be executed on one or more computersystems similar to orchestration system called object-orientatedorchestrator 2000. An object-orientated orchestrator may include twofunctional areas: a data orchestration module 2001 and an artificialintelligence (AI) orchestration module 2002.

In some embodiments, inside data orchestration module 2001, data may betransformed through process module 2003 into datasets with OO-labels.

In some embodiments, inside AI orchestration module 2002, the processingof objects may take place. Firstly, one or more pillars may be selectedbased on business needs (2004). Based on those choices, datasets may beprepared (as shown in block 2005) for use by the pillars. Anorchestration may be composed in module 2006 to create a modelingframework. This modelor may be then bound to one or more datasets inmodule 2007 through the process of binding to create one or morepipelines. In the next step, formed pipelines may be then evaluated inmodule 2008.

FIG. 3 is a diagram that illustrates an exemplary architecture 3000 ofobject-oriented pillar rack in accordance with some of the embodimentsof the present techniques. A scaled propensity modelor 3001 may be amodelor object of the probability of a consumer making an economiccommitment. This scaled propensity is an indicator of inherentcommitment of a customer to a service or product brand. When bound witha dataset, the result modeling object may be computed according tovarious techniques, such as the ones provided in U.S. patent applicationSer. No. 16/127,933, the contents of which are hereby incorporated byreference. A Té modelor 3002 may be used to calibrate the moments intime when specific consumer is likely to engage with specificactivities. The resulting models may be used for churn management ormarketing campaigns. Non-exhaustive examples of encoding are provided inU.S. Patent Application 62/748,287, the contents of which are herebyincorporated by reference. An affinity modelor 3003 may be employed tocapture ranked likes and dislikes of customers for specific items. Theseitems may be, among others, items, services, channels, agents, terms ofcontracts, banking and loans configurations. A best action modelor 3004may be used to create a framework for concurrent KPI compound bestactions at different points in customer journeys. A cluster modelingmodule 3005 may also be used to group customers based on behavior intofinite list for further processing.

FIG. 4 is a flow diagram that illustrates an exemplary OOMtransformation from data to data labeled (ML-label) in accordance withsome of the embodiments of the present techniques. A data transformationmay be done in a data orchestrator rack 4000. In some embodiments, thisrack may be a part of a data orchestration module 2001 shown in FIG. 2 .An ingestion rack 4001 may be the entry point for raw data. In thisrack, some of the following functions may take place: data and schemadrift may be controlled, file headers may be checked, version numbersmay be added to incoming files, data may be routed into clean/errorqueues, and data files may be archived in their raw format.

In some embodiments, a landing rack 4002 may cleanse a 1:1 copy of rawdata. In this rack, some of the following functions may take place:error records may be cleaned, column types may be changed from string tospecific data types, and a version number may be updated.

In some embodiments, a curation rack 4003 may standardize base rawtables. In this rack, some of the following functions may take place:incremental data may be processed, data normalization may be donethrough primary surrogate keys added, de-duplication, referentialintegrity may be checked, data quality may be checked (DQM) throughvalue thresholds and value format, client specific column names may beformed, and the version may be updated.

In some embodiments, a dimensional rack 4004 may manage an analyticaldata warehouse or data lake. In this rack, some of the followingfunctions may take place: data may be encoded in dimensional starschema, column names may be changed from user specific to domainspecific, extension tables as key value stores may be added for userspecific attributes, and the version number may be updated.

In some embodiments, a feature and label bank rack 4005 may extract andengineer features (ML-features) and labels (ML-labels). In this rack,some of the following functions may take place: data may be changed fromdimensional star schema to denormalized flat table, granularity of datamay be adjusted for events, customer-product pairs, and customers, andthe version number may be updated.

Data movements between racks may be controlled through a sheet. Such asheet may signal messages, data, or scripts element moving from racks tobackplane 4007 and messages, data, or scripts element going back toracks 4008.

FIG. 5 is a flow diagram that illustrates an exemplary OOM compositionof object-oriented pillars in accordance with some of the embodiments ofthe present techniques. An orchestration system 5000 may host a librarywith adjudication classes 5001, including:

-   -   a. Sequence: This class of mutators may change a collection of        items into a time sequences for processing.    -   b. Feature: This class may use accessors to gather one or more        ML-feature of a model or modelor, one or more of properties,        features, contexts, ML-state components, OO-state and then use        the features in another model or modelor object.    -   c. Economic optimization: This class may hold one or more        economic objectives and zero or more economic constraints        related to a unitary set of objects (e.g. a person, an product,        a service) or a finite set of unitary set of objects (e.g.        persons and products) or a finite set of unitary sets        complemented by geo-temporal domain (e.g. persons and products        and labor day in Maryland) and uses an allocation algorithm to        maximize the objectives. Examples of objective functions may        include margin optimization, revenue, number of items sold, and        carried interest. Examples of constraints may include Cerebri        Value range, cost of sales, and number of loan officers.        Examples of optimization techniques may include Evolutionary        algorithms, Genetic Algorithm (GA), simulated annealing, TABU        search, harmony search, stochastic hill climbing, particle swarm        optimization, linear programming, dynamic programming, integer        programming, stochastic programming, and shortest path analysis.    -   d. Horses for courses: This class may use accessors to gather        and then analyze different performance measures from the OQM        attributes of modelors and context thereof to select which        modelors out of the set of modelors to use for a specific set of        contexts based on maximize quality value computed from elements        of OQM. This class may also analyze different performance        measures from the OQM attributes of models and context thereof        to select which models out of the set of models to use for a        specific set of contexts based on maximize quality value        computed from elements of OQM.    -   e. Layering: This class may use accessors to gather and then        analyze different measures from the OQM attributes of modelors        and OO-features thereof organized along a semantically preset        taxonomy or ontology to select which performance measures may be        used per OOM-feature for use in a specific set of contexts. This        class may also analyze different measures from the OQM        attributes of models and OO-features thereof organized along a        semantically preset taxonomy or ontology to select which        performance measures should be used per OOM-feature for use in a        specific set of contexts.    -   f. Ensembling: This class may use accessors to gather and then        analyze the outputs and combine the decisions from multiple        models to improve the overall performance.    -   g. Publishing/subscribing: This class may use accessors to        gather relevant attributes and organize them according to        ontologies and mutators using those attributes.

In some embodiments, a pillar composition module 5002 may be used toleverage one or more pillars similar to the pillars shown in FIG. 3 andthe adjudication from module 5001 to develop a modelor (or if bound withdataset, model). Design analysis may determine that the most importantpillar is commitment and who is committed modelor 5003 may be invokedfirst. The second modelor to leverage may be timing module 5004. Thesequence composition 5005 may be used to connect the two. The affinitymodelor 5006 may be invoked next, triggered by message on economicoptimization 5007. The last pillar being invoked may be the how-tomodelor 5008. The final actions may be set in module 5010, messagedthrough module 5011. Modules may perform forward messaging or backwardmessaging to connect with non-adjacent modules. For example, module 5003may message module 5008 through 5012 (e.g., forward messaging) or module5006 may message module 5004 through 5013 (e.g., backward messaging).

FIG. 6 is a flow diagram that illustrates another exemplary pillarcomposition module 5002 for a dual target OOM orchestration ofobject-oriented pillars in accordance with some of the embodiments ofthe present techniques. This flow diagram illustrates the integratedmodeling flow that may facilitate additional improvements with reducedcomplexity that improve performance post original design. The pillarsmay have OQM analysis features, such as feature importance, incrementalcontribution, Shapley information (like Shapley values, or othermeasures of network centrality), Gini impurity, entropy, and crossentropy.

Objective 1 6000 may capture a first business or organizationalobjective. FIG. 6 provides examples related to a marketing campaign of anew product for the sake illustration. Communication between all objectsmay be through messaging. Objective 2 (in box 6001) may capture a secondbusiness or organizational objective. This might be related to an eventcampaign like a sell-athon for the sake of illustration. Design analysismay determine the first pillar to drive the design of Objective 1 iscommitment and who is committed modelor 6002 may be invoked first. Thesecond modelor to leverage may be timing module 6003. Module 6003 maypass messages to the affinity modelor 6004. The last pillar being usedmay be the how-to modelor 6005. There may be a similar flow to supportObjective 2. For example, the first modelor to leverage may be timing6006, messaging the commitment modelor 6007, itself messaging actionmodelor 6008, and the affinity modelor 6009.

The OQM module 6010 inside modelor 6003 may be used to assess thepotential for improvement by leveraging affinity data from ODM module6011 in module 6004 by messaging back and forth 6012.

In some embodiments, this technique may be applied across objectives aswell. The OQM module 6013 inside modelor 6008 may be used to assess thepotential for improvement by timing affinity data from ODM module 6016in module 6003 by messaging back and forth 6015.

Operation research selection of the best actions may take place inmodule 6016. Messages to this module may come from the deepest module inthe Objective 1 flow 6017 or the middle module in the Objective 2 flow6018.

In some embodiments, a governance object 6019 may determine who hasaccess to module 6009. This control may be used to limit access based onthe role of users or persona of a user and what product to sell, forinstance.

FIG. 7 illustrates examples of some of the CEREBRI OOM classes 50 inaccordance with some of the embodiments of the present disclosure.ML-labels are shown in the ML-label class library 7000. (Functionalitydescribed as implemented as libraries may also be implemented inframeworks.) A KPI class 7001 may be used to manage the businessproblems. Two business models may be subscription and purchase. Acustomer class 7002 may capture the business/life-cycle of customerswhether consumers (for B2C) or businesses (for B2B). They may include atrisk customers, or all customers. An item class 7003 may definecommercial items. These items may be physical goods (e.g., cars) orservices (e.g., wireless phone contracts). These items may be rankedhierarchically. That hierarchy or unstructured metadata may be setthrough classes, such as models, options, and customization. In someembodiments, the hierarchy may be a taxonomy hierarchy.

A pillar class library 7005 may include scaled propensity/Cerebri Value7006 (a term which is described in the applications incorporated byreference), timing class 7007, affinity class 7008, and compound bestaction class 7009.

In some embodiments, adjudication classes of modelors or model objectsmay be organized similar to the library shown in block 7010. Not allcompositions may apply to all pillars or KPIs. Modelor compositions andmodel object compositions may include:

-   -   a. Sequence: This class of mutators may change a collection of        items into a time sequences for processing.    -   b. Feature: This class may use accessors to gather one or more        ML-feature of a model or modelor, one or more of properties,        features, contexts, ML-state components, and OO-state and then        use the features in another model or modelor object.    -   c. Economic optimization: This class may hold one or more        economic objectives and zero or more economic constraints        related to a unitary set of objects (typically, but not limited        to, a person, an product, a service) or a finite set of unitary        set of objects (e.g., persons and products) or a finite set of        unitary sets complemented by geo-temporal domain (e.g., persons        and products and labor day in Maryland) and uses an allocation        algorithm to maximize said objectives. Examples of objective        functions include margin optimization, revenue, number of items        sold, and carried interest. Examples of constraints include        Cerebri Value range, cost of sales, and number of loan officers.        Examples of optimization techniques include Evolutionary        algorithms, Genetic Algorithm (GA), simulated annealing, TABU        search, harmony search, stochastic hill climbing, particle swarm        optimization, linear programming, dynamic programming, integer        programming, stochastic programming, and shortest path analysis.    -   d. Horse for courses: This class may use accessors to gather and        then analyze different performance measures from the OQM        attributes of modelors and context thereof to select which        modelors out of the set of modelors to use for a specific set of        contexts based on maximize quality value computed from elements        of OQM. This class may also analyze different performance        measures from the OQM attributes of models and context thereof        to select which models out of the set of models to use for a        specific set of contexts based on maximize quality value        computed from elements of OQM.    -   e. Layering: This class may use accessors to gather and then        analyze different measures from the OQM attributes of modelors        and OO-features thereof organized along a semantically preset        taxonomy or ontology to select which performance measures should        be used per OOM-feature for use in a specific set of contexts.        This class may also analyze different measures from the OQM        attributes of models and OO-features thereof organized along a        semantically preset taxonomy or ontology to select which        performance measures should be used per OOM-feature for use in a        specific set of contexts.    -   f. Ensembling: This class may use accessors to gather and then        analyze the outputs and combine the decisions from multiple        models to improve the overall performance.    -   g. Publishing/subscribing: This class may use accessors to        gather relevant attributes and organize them according to        ontologies and mutators using those attributes.

In some embodiments, modeling methodology classes 7011 may capture someof the key accessors and mutators. Contextualization classes 7012 mayinclude binning (e.g., mapping of continuous attributes into discreteones), winnowing (e.g., reduction of time span, location foci, andbranches in semantic tree), selection of data sources, and selection ofKPIS.

In some embodiments, biding classes 7013 may include binding (or othertype of association) of, for instance, the four types of datasets (e.g.,training, test, validation, and application). The governance classes(7014) may capture the restrictions and business protocols for specificKPIs. They may include OR criteria, operational criteria, actions thatare allowed, and action density (e.g., number of actions per unit time).

In some embodiments, deployment classes 7016 may include realizations7017 including Cerebri Values and numerous KPIs, organized as primaryand secondary. Deployment classes 7016 may also include qualitymeasurements 7018 including data quality monitoring (DQM), model qualitymonitoring (MQM), score quality monitoring (SQM), bias qualitymanagement (BQM), privacy quality management (PQM), and label qualitymonitoring (LQM). Deployment classes 7016 may also include governanceclasses 7019 including support of client model validation using modeldocumentation, CRUD management of items and their metadata, securitycontrol of governance decision maker, QM metric thresholds asconstraints for optimizer, data DQM metric threshold evaluation andanalysis, data DQM metric creep evaluation through data set detection,data lifecycle with gate points and workflow actions, model MQM metricthreshold evaluation and analysis, model MQM metric creep evaluationthrough model drift detection, model output SQM metric thresholdevaluation and analysis, and model lifecycle with gate points andworkflow actions.

FIG. 8 illustrates an example of contextualization and validationimplemented in accordance with some of the embodiments of the presenttechniques. An orchestrator may be processing a composition betweenmodel object modules 8000 which are bound with dataset association 8001and model object 8002 which are bound to dataset association 8003. Theintersection of these two datasets is dataset 8004. The compositionbetween 8000 and 8002 is 8005 and its composition rule are set validatorobject 8006. To ensure the composition is correct, a cross-contextor8007 may be applied to datasets 8002, 8003, and composition rule 8006. Across-contextualization may determine that this composition 8006requires the same dataset association to be used. A new model object8007 may be created by binding the modelor 8008 that is used to createmodel object 8000 with the dataset 8004. A new model object 8009 may becreated by binding the modelor 8010 that is used to create model object8002 with the dataset 8004. Model 8007 may be composed with model 8009using composition 8005 and then composition 8001 to create best actionmodel 8012.

FIG. 9 illustrates an example of a model object field in accordance withsome of the embodiments of the present techniques. An orchestrator 9000may hold the classes 9001 that may have multiple attributes. Attributesmay be divided into multiple buckets, such as object management andlifecycles 9002, governance 9003, and machine learning techniques 9004.

In some embodiments, a modelor collection 10000 may be used to create anoptimized pipeline, may include a series of racks corresponding tonominal steps of a machine learning pipeline as shown in FIG. 10 . Rack10001 may support modelor dealing with data imputation. Rack 10002 maysupport modelors dealing with outliers. Rack 10003 may support modelorsdealing with data augmentation. Rack 10004 may support modelors dealingwith feature enrichments. Rack 10005 may support modelors dealing withthe splitting of datasets. Rack 10006 may support modelors dealing withsampling and balancing datasets. Rack 10007 may support modelors dealingwith feature selections. Rack 10008 may support modelors that performmodeling activities. Rack 10009 may support modelors computingvalidation and scoring. Rack 10010 may support modelors dealing withadjudication heuristics and methodologies. Rack 10011 may supportmodelors implementing operation research. The first five racks may bemonitored by an “Engineering features for the feature label bank” module10013. The sampling 10006 and feature selection 10007 racks may bemonitored by a “creating model ready datasets” module 10014. A modelbuilding rack 10008 may be monitored by a “model building” module 10015.The rest of the racks (e.g. 10009, 10010, and 10011) may be monitored bya “deploying scoring/validation” module 10016. While some may shortcutthrough an optimizer 10017, some of the monitoring modules may passthrough an optimizer 10017 to reduce the search space for optimization.Some of the optimization techniques in 10017 may be nondeterministicpolynomial time (NP) hard. For those, the functional organization ofracks and embedded modelors may reduce the computational load. Agovernance module 10018 may manage the optimization module.

FIG. 11 illustrates an exemplary outcome of an optimization set up inFIG. 10 . A machine learning system 11000 governed by governance module11001 may be resulted into an optimizer 11002 selecting a directed graph11003.

FIG. 12 illustrates an exemplary graphic editor 12001 of a source totarget mapping capabilities facilitated by some of the presenttechniques. Using a graphic user interface, embodiments may facilitateselection of elements 12001 of a STM. The network of STM may be managedthrough workspace pane 12002. Tags needed for building elements may bein panel 12003. In a workspace panel, a user may design sources (12004),mappings (12005), targets (12007), and links 12007 of a directed graphthat may capture their orchestration. UI interactions may be received byevent handlers, which may launch corresponding routines that updateconfigurations (e.g., configuration files) by which the techniquesdescribed herein are directed.

In some embodiments, the OOM framework may be implemented as a libraryinstead or as a hybrid of a library and a framework. In someembodiments, OOM framework may be implemented in a purely-objectoriented programming language, or in a hybrid language with support formultiple paradigms, e.g., also supporting functional programming orimperative programing support. The OOM framework may be implemented as acompiled or as an interpreted language, e.g., for the latter, with aninterpreter that interprets source code to bytecode, which can becompiled to machine code and executed via a suitable virtual machine fora host computing system.

The OOM framework may organize database structure, program code, andprogram state in objects. Objects may include both methods (which mayalso be referred to as routines, procedures, or functions), andattributes, which may be data of the object. Objects may be instances ofclasses supported by the OOM framework, e.g., instance A of class 1 andinstance B of class 1 may both be objects with the same set of aplurality of methods and attributes, and after, or as part ofinstantiations, the values taken by the attributes may evolve and differbetween the object that is instance A and the object that is instance B,e.g., instance A and instance B may be independent from one another. Insome cases, classes may support class variables that are accessible toobjects that are instances of the respective class. In some cases,methods of objects may have access (e.g., to read, write, or both readand write) to attributes of the respective object, but not to attributesof other objects. In some cases, some attributes of some objects may bedesignated as public or private to modulate access by methods otherobjects. In some cases, objects may receive messages, like invocationsof their methods from other objects, and the called object may determinewhich code to execute in response to the invocation, e.g., with dynamicdispatch, and in some cases, different instances of a class may beconfigured to dispatch to different code for the same invocationmessage. In some cases, objects may be composed of other objects ofother classes, and in some cases, classes may indicate inheritance toorganize objects according to a hierarchical ontology of types, e.g.,where sub-types inherit the attributes and methods of the types fromwhich they inherit. In some cases, the OOM framework may supportpolymorphism, and objects may be configured to operate on a type ofobjects, its sub-types, or its sub-sub-types.

In some cases, designs in the OOM framework may be implemented withvarious design pattern. For example, the following creation patterns maybe used: Factory method pattern, Abstract factory pattern, Singletonpattern, Builder pattern, or Prototype pattern. In some cases, thefollowing structural patterns may be used: Adapter pattern, Bridgepattern, Composite pattern, Decorator pattern, Facade pattern, Flyweightpattern, or Proxy pattern. In some cases, the following behaviorpatterns may be used: Chain-of-responsibility pattern, Command pattern,Interpreter pattern, Iterator pattern, Mediator pattern, Mementopattern, Observer pattern, State pattern, Strategy pattern, Templatemethod pattern, or Visitor pattern.

In some cases, the OOM framework specifies a set of symbols to modelvarious aspects of creating and using machine-learning models. In somecases, the OOM framework specifies an ontology of classes (which may behierarchical), with different branches or hierarchies pertaining todifferent areas of concern in the life cycle of machine learning models,like those discussed above.

In some embodiments, the controller 10 (e.g., an object-orientedmodeling module thereof) may execute a process 200 shown in FIG. 13 . Insome embodiments, different subsets of this process 200 may be executedby the illustrated components of the controller 10, so those featuresare described herein concurrently. It should be emphasized, though, thatembodiments of the process 200 are not limited to implementations withthe architecture of FIG. 1 , and that the architecture of FIG. 1 mayexecute processes different from that described with reference to FIG.13 , none of which is to suggest that any other description herein islimiting.

In some embodiments, the process 200 (and other functionality herein)may be implemented with program code or other instructions stored on atangible, non-transitory, machine-readable medium, such that when theinstructions are executed by one or more processors (a term which asused herein refers to physical processors, e.g., implemented on asemiconductor device), the described functionality is effectuated. Insome embodiments, notwithstanding use of the singular term “medium,” themedium may be distributed, with different subsets of the instructionsstored on different computing devices that effectuate those differentsubsets, an arrangement consistent with use of the singular term“medium” along with monolithic applications on a single device. In someembodiments, the described operations may be executed in a differentorder, some or all of the operations may be executed multiple times,operations may be executed concurrently with one another or multipleinstances of the described process, additional operations may beinserted, operations may be omitted, operations may be executedserially, or the processes described may otherwise be varied, again noneof which is to suggest that any other description herein is limiting.

In some embodiments, the process 200 includes obtaining, as indicated byblock 202, for a plurality of entities, datasets. The datasets may beevents or attributes involving the entities. In some embodiments, atleast a subset of the events are actions by the entities and some ofthese actions may be targeted actions.

In some embodiments, a plurality of objects may be formed, as indicatedby block 204. A data domain may be used to transform datasets intoobjects. Objects may include events and attributes of entities(attributes of entities and events may be a type of attribute of anobject) extracted from datasets.

In some embodiments, these datasets may be labeled with object-orientedtags, as indicated by block 206, used to classify entity logs. In someembodiments, labelling is a process of adding tags to data. Augmentingdata with labels may make it more informative and more manageable. Oneuse of labelling in object-oriented modeling may be management oflabels, not as elements in a list or table, but as objects allowing acollection of tags to be used. As objects, OO-labels may be managed andorganized through enforced set of grammar and semantic rules. Oneattribute of an OO-label may be a user facing text for sake of userexperience (UX). In some embodiments, OO-labels may be used in a singleontology used for multiple customers, each with a ML-label for theirbusiness realizing a MUPL. OO-labels may encode meta-data about adataset, control information, and governance information among otherthings.

In some embodiments, a library of classes may be formed, as indicated byblock 208. The library of classes may include the classes shown in FIG.7 . Also, a plurality of object-manipulation functions may be formed asindicated by block 210. Each of the object-manipulation objects may beconfigured to leverage a specific class.

In some embodiments, an object-orientated orchestration may be formed asindicated by block 212. An orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system. Orchestrations may operate on objects to create newobjects and thus may be recursive.

In some embodiments, the object-orientated orchestration may then beused to select a set of actions to achieve a given targeted actionsimilar to the process shown in FIG. 2 . To this end, some embodimentsmay receive a request from an entity or a subscriber as indicated byblock 214. Some embodiments may then determine the set of actions toachieve (or increase the likelihood of achieving) the given targetedaction, as indicated by block 216 in FIG. 13 , for instance, with apotential targeted action 20 in FIG. 1 .

In some embodiments, the controller 10 may execute a process 300 shownin FIG. 14 . In some embodiments, different subsets of this process 300may be executed by the illustrated components of the controller 10, sothose features are described herein concurrently. It should beemphasized, though, that embodiments of the process 300 are not limitedto implementations with the architecture of FIG. 1 , and that thearchitecture of FIG. 1 may execute processes different from thatdescribed with reference to FIG. 14 , none of which is to suggest thatany other description herein is limiting.

In some embodiments, the process 300 includes obtaining, as indicated byblock 302, for a plurality of entities, datasets. The datasets may beevents or attributes involving the entities. In some embodiments, atleast a subset of the events are actions by the entities and some ofthese actions may be targeted actions. The actions may include exogenousactions and endogenous actions.

In some embodiments, a plurality of objects may be formed, as indicatedby block 304. In some embodiments, all of the datasets are transferredinto objects. In some embodiments, only a portion of the datasets aretransferred into objects. Objects may include events and attributesextracted from datasets, both of which may be attributes of therespective object.

In some embodiments, these datasets may be labeled with object-orientedtags, as indicated by block 306, used to classify entity logs. In someembodiments, labelling is a process of adding tags to data.

In some embodiments, a library of classes may be formed, as indicated byblock 308. The library of classes may include the classes shown in FIG.7 . Also, a plurality of object-manipulation functions may be formed asindicated by block 310. Each of the object-manipulation objects may beconfigured to leverage a specific class.

In some embodiments, an object-orientated orchestration may be formed asindicated by block 312. An orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system. Orchestrations may operate on objects to create newobjects and thus may be recursive.

In some embodiments, a request may be received from an entity (e.g.,another object or other body of executing code) or a subscriber asindicated by block 314 to determine a set of actions required to achievea specific targeted action specified by the request. Some embodimentsmay then determine the set of actions to achieve (a term used broadly toalso refer to increasing the likelihood of achieving) the given targetedaction, using a compiler function as indicated by block 316 in FIG. 14 ,for instance, with a potential targeted action 20 in FIG. 1 . A compilerfunction may first pick a set of classes based on the specific targetedaction and then use a series of object-manipulation functions todetermine the set of actions, needed to be done by the entity or otherentities, to achieve the specific targeted action.

In some embodiments, the controller 10 may execute a process 400 shownin FIG. 15 . In some embodiments, different subsets of this process 400may be executed by the illustrated components of the controller 10, sothose features are described herein concurrently. It should beemphasized, though, that embodiments of the process 400 are not limitedto implementations with the architecture of FIG. 1 , and that thearchitecture of FIG. 1 may execute processes different from thatdescribed with reference to FIG. 15 , none of which is to suggest thatany other description herein is limiting.

In some embodiments, the process 400 includes first obtaining, asindicated by block 402, for a plurality of entities, datasets.Thereafter, multiple objects may be formed, as indicated by block 404,and the datasets may be labeled with object-oriented tags, as indicatedby block 406, used to classify entity logs.

In some embodiments, a library of classes may be formed, as indicated byblock 408. The library of classes may include the classes shown in FIG.7 . Also, a plurality of object-manipulation functions may be formed asindicated by block 410. Each of the object-manipulation objects may beconfigured to leverage a specific class.

In some embodiments, a training dataset may be formed using the datasetsof the plurality of entities, as shown by block 412. In someembodiments, the training dataset may describe scenarios that haveoccurred in the past. In some embodiments, the training dataset maydescribe scenarios that have not occurred, and thus are virtual. Use ofthe terms “form” and “generate” are used broadly and use of differentterms should not be read to necessarily refer to different operations invirtue of using different terminology, as both of these terms generallyinclude causing the described thing to come into being, whether bymodifying an existing thing or forming a new copy or instance.

Next, some embodiments may input the training datasets into a machinelearning model model that indicates interdependency of the plurality ofobject-manipulation functions in leveraging a specific class, asindicated by block 414 in FIG. 15 . In some embodiments, the trainedmodel may, in response, output a score indicative of interdependency ofthe plurality of object-manipulation functions. In some cases, theoutput may be one of the scores or values described in patentapplication Ser. No. 15/456,059, titled BUSINESS ARTIFICIAL INTELLIGENCEMANAGEMENT ENGINE, the contents of which are hereby incorporated byreference.

In some of the embodiments related to machine learning, new features(which may increase dimension of datasets) may also be obtained from oldfeatures using a method known as feature engineering using knowledgeabout a problem to create new variables. In some embodiments, whilefeature engineering may be part of the application of machine learning,it may be difficult and expensive in terms of time, resource, and talentperspective. In some embodiments, CEREBRI may useobject-publish-subscribe modeling to convey improvements in featureengineering from one object to another.

In some embodiments related to machine learning, features that may notbe helpful in the performance of the models may be removed (which maydecrease in dimensionality of datasets); this is an example of what isreferred to as feature selection. Feature selection may be helpful interms of the required time, resource, and talent perspective. In someembodiments, CEREBRI may use object-publish-subscribe modeling to conveyimprovements in feature selection from one object to another.

In some embodiments, when an object is a model or modelor, it may embeda machine learning state that is ML-State. The state of the object maybe to a large degree a superset the machine learning state of a modelthat captures latent, Markov, reward, quality, or governanceinformation.

Different types of training may be applied depending upon the type ofmodel in use. Any of the types of models described above may be applied.In some embodiments, the model is policy in a reinforcement learningmodel. In some embodiments, the model is a classifier configured toclassify object classes. Various types of processing may be performed bymachine learning model on the datasets, including the processes shown inFIG. 10 .

In some embodiments, multiple interdependency graphs are formed, asshown by block 416, which can track the relationship between differentclasses.

Some embodiments may store the resulting model in memory, as indicatedby block 418. As noted, trained models may be expressed as a lookuptable mapping inputs to outputs, sets of values for constants orvariables in software routines, as values of parameters in closed-formequations, or combinations thereof.

In some embodiments, an object-orientated orchestration may be formed asindicated by block 420. An orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system.

In some embodiments, the object-orientated orchestration may then beused to select a set of actions to achieve a given targeted actionsimilar to the process shown in FIG. 2 . To this end, some embodimentsmay receive a request from an entity or a subscriber as indicated byblock 422. Some embodiments may then determine the set of actions toachieve (or increase the likelihood of achieving) the given targetedaction, as indicated by block 424 in FIG. 135 for instance, with apotential targeted action 20 in FIG. 1 .

In some embodiments, the controller 10 may execute a process 500 shownin FIG. 16 . In some embodiments, different subsets of this process 500may be executed by the illustrated components of the controller 10, sothose features are described herein concurrently. It should beemphasized, though, that embodiments of the process 500 are not limitedto implementations with the architecture of FIG. 1 , and that thearchitecture of FIG. 1 may execute processes different from thatdescribed with reference to FIG. 16 , none of which is to suggest thatany other description herein is limiting.

In some embodiments, the process 500 (and other functionality herein)may be implemented with program code or other instructions stored on atangible, non-transitory, machine-readable medium, such that when theinstructions are executed by one or more processors (a term which asused herein refers to physical processors, e.g., implemented on asemiconductor device), the described functionality is effectuated. Insome embodiments, notwithstanding use of the singular term “medium,” themedium may be distributed, with different subsets of the instructionsstored on different computing devices that effectuate those differentsubsets, an arrangement consistent with use of the singular term“medium” along with monolithic applications on a single device. In someembodiments, the described operations may be executed in a differentorder, some or all of the operations may be executed multiple times,operations may be executed concurrently with one another or multipleinstances of the described process, additional operations may beinserted, operations may be omitted, operations may be executedserially, or the processes described may otherwise be varied, again noneof which is to suggest that any other description herein is limiting.

In some embodiments, the process 500 includes obtaining, as indicated byblock 502, for a plurality of entities, datasets. The datasets may beevents or attributes involving the entities. In some embodiments, atleast a subset of the events are actions by the entities and some ofthese actions may be targeted actions.

In some embodiments, a plurality of objects may be formed, as indicatedby block 504. Objects may include events and attributes extracted fromdatasets.

In some embodiments, these datasets may be labeled with object-orientedtags, as indicated by block 506, used to classify entity logs. In someembodiments, labelling is a process of adding tags to data. Augmentingdata with labels may make it more informative and more manageable. Oneuse of labelling in object-oriented modeling may be management oflabels, not as elements in a list or table, but as objects allowing acollection of tags to be used. As objects, OO-labels may be managed andorganized through enforced set of grammar and semantic rules. Oneattribute of an OO-label may be a user facing text for sake of userexperience (UX). In some embodiments, OO-labels may be used in a singleontology used for multiple customers, each with a ML-label for theirbusiness realizing a MUPL. OO-labels may encode meta-data about adataset, control information, and governance information among otherthings.

In some embodiments, a library of classes may be formed, as indicated byblock 508. The library of classes may include the classes shown in FIG.7 . Some of the classes of the library of classes may include variousquality management systems, as indicated by block 510.

In some embodiments, validators may be CEREBRI public objects ensuring,integrity, governance, and quality objects used to ensure no datalinkage or inconsistency of datasets, ML-labels, performance (OQM) andwindows of processing. Validators may check database consistency appliedto aspects of an OOM. A function of validators may be triggeringretraining of part or complete pipelines based on quality or operationaltriggers.

In some embodiments, quality Management (QM) in an object-orientedmodeling paradigm that may be implemented as a process that integratesraw data ingestion, manipulation, transformation, composition, andstorage for building artificial intelligence models. In legacy designs,quality management may include (and in some cases may be limited to)Extract, Transform, and Load (ETL) phases of effort and to the reportingof model performance (e.g., recall, precision, F1, etc.) from an end toend perspective as a quality.

Deposition of design process and operation of models, developed usingOOM into objects, may facilitate efforts to cause quality to be embeddedin objects. Quality may be attributes of objects. Modelor, boundmodelors, and pipelines may be managed through multiple lifecyclesrather than a single one. In some embodiments, Object-Oriented QM (OQM)may have six components:

-   -   a. Data quality monitoring (DQM): DQM measures, not necessarily        exclusively (which is not to suggest that other lists are        exclusive), new or missing data source (table) or data element,        counts, mull count and unique counts, and datatype changes. DQM        may be used to figure out which data sources are reliable.    -   b. Model quality monitoring (MQM): MQM may measure, not        necessarily exclusively (which is not to suggest that other        lists are exclusive), model-based metrics, such as F1,        precision, recall, etc., or data, and triggers retraining for        drift.    -   c. Score quality monitoring (SQM): SQM may perform model        hypothesis tests, including Welch's t-test (e.g., parametric        test for equal means) and the Mann-Whitney U-test (e.g.,        non-parametric test for equal distributions). SQM may also        compute lift tables, a decile table based on the predicted        probability of positive class membership, with the cumulative        distribution function of positive cases added in. The gain chart        is a plot of the cumulative distribution function of positive        cases may be included as a function of quantile.    -   d. Label quality monitoring (LQM): Labels may be categorical and        bound by semantic rules or ontologies. LQM may be used to        understand which data sources are leverageable and impactful.        LQM may be used for data debt management and enhancing        compositions for performance.    -   e. Bias quality monitoring (BQM): Bias is a systematic        distortion of the relationship between a variable, a data set,        and results. Three types of bias may be distinguished:        information bias, selection bias, and confounding, in some        embodiments.    -   f. Private quality monitoring (PQM): Privacy may cover        personally identifiable information and access of privileged        information.

In some embodiments, a plurality of object-manipulation functions may beformed as indicated by block 512. Each of the object-manipulationobjects may be configured to leverage a specific class.

In some embodiments, an object-orientated orchestration may be formed asindicated by block 514. An orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system. Orchestrations may operate on objects to create newobjects and thus may be recursive.

In some embodiments, the object-orientated orchestration may then beused to select a set of actions to achieve a given targeted action. Tothis end, some embodiments may receive a request from an entity or asubscriber as indicated by block 516. Some embodiments may thendetermine the set of actions to achieve (or increase the likelihood ofachieving) the given targeted action, as indicated by block 518 in FIG.16 , while ensuring a certain level of accuracy using the qualitymanagement systems.

In some embodiments, the controller 10 may execute a process 600 shownin FIG. 17 . In some embodiments, different subsets of this process 600may be executed by the illustrated components of the controller 10, sothose features are described herein concurrently. It should beemphasized, though, that embodiments of the process 600 are not limitedto implementations with the architecture of FIG. 1 , and that thearchitecture of FIG. 1 may execute processes different from thatdescribed with reference to FIG. 17 , none of which is to suggest thatany other description herein is limiting.

In some embodiments, the process 600 includes obtaining, as indicated byblock 602, for a plurality of entities, datasets. The datasets may beevents or attributes involving the entities. In some embodiments, atleast a subset of the events are actions by the entities and some ofthese actions may be targeted actions. Some of the attributes aregovernance attributes, including the governance attributes 9003 shown inFIG. 9 .

In some embodiments, governance in CEREBRI may be a set of structures,processes and policies by which pipeline development, deployment, anduse function within an organization or set of organizations is directed,managed, and controlled to yield business value and to mitigate risk.

In some embodiments, a policy in CEREBRI may refer to set of rules,controls, and resolutions put in place to dictate model behaviorindividually as a whole. Policies are a way governance is encoded andmanaged in some embodiments. The policy items are referred to aspolicors and implemented as CEREBRI objects. As the number of rulesincrease, a policy-driven OOM may suffer from inconsistencies incurredby contradicting rules governing its behavior. In some embodiments,meta-policies (e.g., detection rules) may be used for the detection ofconflicts. In some embodiments, policy relationships may be used. Insome embodiments, attribute-based applicability spaces may be used.

In some embodiments, a plurality of objects may be formed, as indicatedby block 604. A data domain may be used to transform datasets intoobjects. Objects may include events and attributes extracted fromdatasets.

In some embodiments, these datasets may be labeled with object-orientedtags, as indicated by block 606, used to classify entity logs. Then, alibrary of classes may be formed, as indicated by block 608. The libraryof classes may include the classes shown in FIG. 7 . Also, a pluralityof object-manipulation functions may be formed as indicated by block610. Each of the object-manipulation objects may be configured toleverage a specific class.

In some embodiments, an object-orientated orchestration may be formed asindicated by block 612. An orchestration may be an organization of acollection of steps that perform a specific objective when executed by acomputing system. Orchestrations may operate on objects to create newobjects and thus may be recursive.

In some embodiments, the object-orientated orchestration may then beused to select a set of actions to achieve a given targeted actionsimilar to the process shown in FIG. 2 . To this end, some embodimentsmay receive a request from an entity or a subscriber as indicated byblock 614. Some embodiments may then determine the set of actions toachieve (or increase the likelihood of achieving) the given targetedaction, as indicated by block 616 in FIG. 17 , for instance, with apotential targeted action 20 in FIG. 1 .

The physical architecture may take a variety of forms, including asmonolithic on-premises applications executing on a single host on asingle computing device, distributed on-premise applications executingon multiple hosts on one or more local computing devices on a privatenetwork, distributed hybrid applications having on-premises componentsand other components provided in a software as a service (SaaS)architecture hosted in a remote data center with multi-tenancy accessedvia the network, and distributed SaaS implementations in which varioussubsets or all of the functionality described herein is implemented on acollection of computing devices in one or more remote data centersserving multiple tenants, each accessing the hosted functionality underdifferent tenant accounts. In some embodiments, the computing devicesmay take the form of the computing device described below with referenceto FIG. 18 .

FIG. 18 is a diagram that illustrates an exemplary computing system 1000in accordance with embodiments of the present technique. Variousportions of systems and methods described herein, may include or beexecuted on one or more computer systems similar to computing system1000. Further, processes and modules described herein may be executed byone or more processing systems similar to that of computing system 1000.

Computing system 1000 may include one or more processors (e.g.,processors 1010 a-1010 n) coupled to system memory 1020, an input/outputI/O device interface 1030, and a network interface 1040 via aninput/output (I/O) interface 1050. A processor may include a singleprocessor or a plurality of processors (e.g., distributed processors). Aprocessor may be any suitable processor capable of executing orotherwise performing instructions. A processor may include a centralprocessing unit (CPU) that carries out program instructions to performthe arithmetical, logical, and input/output operations of computingsystem 1000. A processor may execute code (e.g., processor firmware, aprotocol stack, a database management system, an operating system, or acombination thereof) that creates an execution environment for programinstructions. A processor may include a programmable processor. Aprocessor may include a Graphic Processing Unit (GPU). A processor mayinclude general or special purpose microprocessors. A processor mayreceive instructions and data from a memory (e.g., system memory 1020).Computing system 1000 may be a uni-processor system including oneprocessor (e.g., processor 1010 a), or a multi-processor systemincluding any number of suitable processors (e.g., 1010 a-1010 n).Multiple processors may be employed to provide for parallel orsequential execution of one or more portions of the techniques describedherein. Processes, such as logic flows, described herein may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating corresponding output. Processes described herein may beperformed by, and apparatus can also be implemented as, special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or anASIC (application specific integrated circuit). Computing system 1000may include a plurality of computing devices (e.g., distributed computersystems) to implement various processing functions.

I/O device interface 1030 may provide an interface for connection of oneor more I/O devices 1060 to computer system 1000. I/O devices mayinclude devices that receive input (e.g., from a user) or outputinformation (e.g., to a user). I/O devices 1060 may include, forexample, graphical user interface presented on displays (e.g., a cathoderay tube (CRT) or liquid crystal display (LCD) monitor), pointingdevices (e.g., a computer mouse or trackball), keyboards, keypads,touchpads, scanning devices, voice recognition devices, gesturerecognition devices, printers, audio speakers, microphones, cameras, orthe like. I/O devices 1060 may be connected to computer system 1000through a wired or wireless connection. I/O devices 1060 may beconnected to computer system 1000 from a remote location. I/O devices1060 located on remote computer system, for example, may be connected tocomputer system 1000 via a network and network interface 1040.

Network interface 1040 may include a network adapter that provides forconnection of computer system 1000 to a network. Network interface 1040may facilitate data exchange between computer system 1000 and otherdevices connected to the network. Network interface 1040 may supportwired or wireless communication. The network may include an electroniccommunication network, such as the Internet, a local area network (LAN),a wide area network (WAN), a cellular communications network, or thelike.

System memory 1020 may be configured to store program instructions 1100or data 1110. Program instructions 1100 may be executable by a processor(e.g., one or more of processors 1010 a-1010 n) to implement one or moreembodiments of the present techniques. Instructions 1100 may includemodules of computer program instructions for implementing one or moretechniques described herein with regard to various processing modules.Program instructions may include a computer program (which in certainforms is known as a program, software, software application, script, orcode). A computer program may be written in a programming language,including compiled or interpreted languages, or declarative orprocedural languages. A computer program may include a unit suitable foruse in a computing environment, including as a stand-alone program, amodule, a component, or a subroutine. A computer program may or may notcorrespond to a file in a file system. A program may be stored in aportion of a file that holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub programs, or portionsof code). A computer program may be deployed to be executed on one ormore computer processors located locally at one site or distributedacross multiple remote sites and interconnected by a communicationnetwork.

System memory 1020 may include a tangible program carrier having programinstructions stored thereon. A tangible program carrier may include anon-transitory computer readable storage medium. A non-transitorycomputer readable storage medium may include a machine-readable storagedevice, a machine-readable storage substrate, a memory device, or anycombination thereof. Non-transitory computer readable storage medium mayinclude non-volatile memory (e.g., flash memory, ROM, PROM, EPROM,EEPROM memory), volatile memory (e.g., random access memory (RAM),static random access memory (SRAM), synchronous dynamic RAM (SDRAM)),bulk storage memory (e.g., CD-ROM and/or DVD-ROM, hard-drives), or thelike. System memory 1020 may include a non-transitory computer readablestorage medium that may have program instructions stored thereon thatare executable by a computer processor (e.g., one or more of processors1010 a-1010 n) to cause the subject matter and the functional operationsdescribed herein. A memory (e.g., system memory 1020) may include asingle memory device and/or a plurality of memory devices (e.g.,distributed memory devices). Instructions or other program code toprovide the functionality described herein may be stored on a tangible,non-transitory computer readable media. In some cases, the entire set ofinstructions may be stored concurrently on the media, or in some cases,different parts of the instructions may be stored on the same media atdifferent times.

I/O interface 1050 may be configured to coordinate I/O traffic betweenprocessors 1010 a-1010 n, system memory 1020, network interface 1040,I/O devices 1060, and/or other peripheral devices. I/O interface 1050may perform protocol, timing, or other data transformations to convertdata signals from one component (e.g., system memory 1020) into a formatsuitable for use by another component (e.g., processors 1010 a-1010 n).I/O interface 1050 may include support for devices attached throughvarious types of peripheral buses, such as a variant of the PeripheralComponent Interconnect (PCI) bus standard or the Universal Serial Bus(USB) standard.

Embodiments of the techniques described herein may be implemented usinga single instance of computer system 1000 or multiple computer systems1000 configured to host different portions or instances of embodiments.Multiple computer systems 1000 may provide for parallel or sequentialprocessing/execution of one or more portions of the techniques describedherein.

Those skilled in the art will appreciate that computer system 1000 ismerely illustrative and is not intended to limit the scope of thetechniques described herein. Computer system 1000 may include anycombination of devices or software that may perform or otherwise providefor the performance of the techniques described herein. For example,computer system 1000 may include or be a combination of acloud-computing system, a data center, a server rack, a server, avirtual server, a desktop computer, a laptop computer, a tabletcomputer, a server device, a client device, a mobile telephone, apersonal digital assistant (PDA), a mobile audio or video player, a gameconsole, a vehicle-mounted computer, or a Global Positioning System(GPS), or the like. Computer system 1000 may also be connected to otherdevices that are not illustrated, or may operate as a stand-alonesystem. In addition, the functionality provided by the illustratedcomponents may in some embodiments be combined in fewer components ordistributed in additional components. Similarly, in some embodiments,the functionality of some of the illustrated components may not beprovided or other additional functionality may be available.

Those skilled in the art will also appreciate that while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memory,elements of distributed systems, and other storage devices for purposesof memory management and data integrity. Alternatively, in otherembodiments some or all of the software components may execute in memoryon another device and communicate with the illustrated computer systemvia inter-computer communication. Some or all of the system componentsor data structures may also be stored (e.g., as instructions orstructured data) on a computer-accessible medium or a portable articleto be read by an appropriate drive, various examples of which aredescribed above. In some embodiments, instructions stored on acomputer-accessible medium separate from computer system 1000 may betransmitted to computer system 1000 via transmission media or signalssuch as electrical, electromagnetic, or digital signals, conveyed via acommunication medium such as a network or a wireless link. Variousembodiments may further include receiving, sending, or storinginstructions or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Accordingly, the presenttechniques may be practiced with other computer system configurations.

In block diagrams, illustrated components are depicted as discretefunctional blocks, but embodiments are not limited to systems in whichthe functionality described herein is organized as illustrated. Thefunctionality provided by each of the components may be provided bysoftware or hardware modules that are differently organized than ispresently depicted, for example such software or hardware may beintermingled, conjoined, replicated, broken up, distributed (e.g. withina data center or geographically), or otherwise differently organized.The functionality described herein may be provided by one or moreprocessors of one or more computers executing code stored on a tangible,non-transitory, machine readable medium. In some cases, notwithstandinguse of the singular term “medium,” the instructions may be distributedon different storage devices associated with different computingdevices, for instance, with each computing device having a differentsubset of the instructions, an implementation consistent with usage ofthe singular term “medium” herein. In some cases, third party contentdelivery networks may host some or all of the information conveyed overnetworks, in which case, to the extent information (e.g., content) issaid to be supplied or otherwise provided, the information may beprovided by sending instructions to retrieve that information from acontent delivery network.

The reader should appreciate that the present application describesseveral independently useful techniques. Rather than separating thosetechniques into multiple isolated patent applications, applicants havegrouped these techniques into a single document because their relatedsubject matter lends itself to economies in the application process. Butthe distinct advantages and aspects of such techniques should not beconflated. In some cases, embodiments address all of the deficienciesnoted herein, but it should be understood that the techniques areindependently useful, and some embodiments address only a subset of suchproblems or offer other, unmentioned benefits that will be apparent tothose of skill in the art reviewing the present disclosure. Due to costsconstraints, some techniques disclosed herein may not be presentlyclaimed and may be claimed in later filings, such as continuationapplications or by amending the present claims. Similarly, due to spaceconstraints, neither the Abstract nor the Summary of the Inventionsections of the present document should be taken as containing acomprehensive listing of all such techniques or all aspects of suchtechniques.

It should be understood that the description and the drawings are notintended to limit the present techniques to the particular formdisclosed, but to the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the present techniques as defined by the appended claims.Further modifications and alternative embodiments of various aspects ofthe techniques will be apparent to those skilled in the art in view ofthis description. Accordingly, this description and the drawings are tobe construed as illustrative only and are for the purpose of teachingthose skilled in the art the general manner of carrying out the presenttechniques. It is to be understood that the forms of the presenttechniques shown and described herein are to be taken as examples ofembodiments. Elements and materials may be substituted for thoseillustrated and described herein, parts and processes may be reversed oromitted, and certain features of the present techniques may be utilizedindependently, all as would be apparent to one skilled in the art afterhaving the benefit of this description of the present techniques.Changes may be made in the elements described herein without departingfrom the spirit and scope of the present techniques as described in thefollowing claims. Headings used herein are for organizational purposesonly and are not meant to be used to limit the scope of the description.

As used throughout this application, the word “may” is used in apermissive sense (i.e., meaning having the potential to), rather thanthe mandatory sense (i.e., meaning must). The words “include”,“including”, and “includes” and the like mean including, but not limitedto. As used throughout this application, the singular forms “a,” “an,”and “the” include plural referents unless the content explicitlyindicates otherwise. Thus, for example, reference to “an element” or “aelement” includes a combination of two or more elements, notwithstandinguse of other terms and phrases for one or more elements, such as “one ormore.” The term “or” is, unless indicated otherwise, non-exclusive,i.e., encompassing both “and” and “or.” Terms describing conditionalrelationships, e.g., “in response to X, Y,” “upon X, Y,”, “if X, Y,”“when X, Y,” and the like, encompass causal relationships in which theantecedent is a necessary causal condition, the antecedent is asufficient causal condition, or the antecedent is a contributory causalcondition of the consequent, e.g., “state X occurs upon condition Yobtaining” is generic to “X occurs solely upon Y” and “X occurs upon Yand Z.” Such conditional relationships are not limited to consequencesthat instantly follow the antecedent obtaining, as some consequences maybe delayed, and in conditional statements, antecedents are connected totheir consequents, e.g., the antecedent is relevant to the likelihood ofthe consequent occurring. Statements in which a plurality of attributesor functions are mapped to a plurality of objects (e.g., one or moreprocessors performing steps A, B, C, and D) encompasses both all suchattributes or functions being mapped to all such objects and subsets ofthe attributes or functions being mapped to subsets of the attributes orfunctions (e.g., both all processors each performing steps A-D, and acase in which processor 1 performs step A, processor 2 performs step Band part of step C, and processor 3 performs part of step C and step D),unless otherwise indicated. Similarly, reference to “a computer system”performing step A and “the computer system” performing step B caninclude the same computing device within the computer system performingboth steps or different computing devices within the computer systemperforming steps A and B. Further, unless otherwise indicated,statements that one value or action is “based on” another condition orvalue encompass both instances in which the condition or value is thesole factor and instances in which the condition or value is one factoramong a plurality of factors. Unless otherwise indicated, statementsthat “each” instance of some collection have some property should not beread to exclude cases where some otherwise identical or similar membersof a larger collection do not have the property, i.e., each does notnecessarily mean each and every. Limitations as to sequence of recitedsteps should not be read into the claims unless explicitly specified,e.g., with explicit language like “after performing X, performing Y,” incontrast to statements that might be improperly argued to imply sequencelimitations, like “performing X on items, performing Y on the X'editems,” used for purposes of making claims more readable rather thanspecifying sequence. Statements referring to “at least Z of A, B, andC,” and the like (e.g., “at least Z of A, B, or C”), refer to at least Zof the listed categories (A, B, and C) and do not require at least Zunits in each category. Unless specifically stated otherwise, asapparent from the discussion, it is appreciated that throughout thisspecification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining” or the like refer to actionsor processes of a specific apparatus, such as a special purpose computeror a similar special purpose electronic processing/computing device.Features described with reference to geometric constructs, like“parallel,” “perpendicular/orthogonal,” “square”, “cylindrical,” and thelike, should be construed as encompassing items that substantiallyembody the properties of the geometric construct, e.g., reference to“parallel” surfaces encompasses substantially parallel surfaces. Thepermitted range of deviation from Platonic ideals of these geometricconstructs is to be determined with reference to ranges in thespecification, and where such ranges are not stated, with reference toindustry norms in the field of use, and where such ranges are notdefined, with reference to industry norms in the field of manufacturingof the designated feature, and where such ranges are not defined,features substantially embodying a geometric construct should beconstrued to include those features within 15% of the definingattributes of that geometric construct. The terms “first”, “second”,“third,” “given” and so on, if used in the claims, are used todistinguish or otherwise identify, and not to show a sequential ornumerical limitation. As is the case in ordinary usage in the field,data structures and formats described with reference to uses salient toa human need not be presented in a human-intelligible format toconstitute the described data structure or format, e.g., text need notbe rendered or even encoded in Unicode or ASCII to constitute text;images, maps, and data-visualizations need not be displayed or decodedto constitute images, maps, and data-visualizations, respectively;speech, music, and other audio need not be emitted through a speaker ordecoded to constitute speech, music, or other audio, respectively.Computer implemented instructions, commands, and the like are notlimited to executable code and can be implemented in the form of datathat causes functionality to be invoked, e.g., in the form of argumentsof a function or API call. To the extent bespoke noun phrases (and othercoined terms) are used in the claims and lack a self-evidentconstruction, the definition of such phrases may be recited in the claimitself, in which case, the use of such bespoke noun phrases should notbe taken as invitation to impart additional limitations by looking tothe specification or extrinsic evidence.

In this patent, to the extent any U.S. patents, U.S. patentapplications, or other materials (e.g., articles) have been incorporatedby reference, the text of such materials is only incorporated byreference to the extent that no conflict exists between such materialand the statements and drawings set forth herein. In the event of suchconflict, the text of the present document governs, and terms in thisdocument should not be given a narrower reading in virtue of the way inwhich those terms are used in other materials incorporated by reference.

The present techniques will be better understood with reference to thefollowing enumerated embodiments:

-   -   1A. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: obtaining, with one or more        processors, for a plurality of entities, datasets, wherein: the        datasets comprise a plurality of entity logs; the entity logs        comprise events involving the entities; at least some of the        events are actions by the entities; at least some of the actions        are targeted actions; the entity logs comprise or are otherwise        associated with attributes of the entities; and the events are        distinct from the attributes; orchestrating, with one or more        processors, an object-orientated application or service by:        forming a plurality of objects, wherein each object of the        plurality of objects comprises a different set of attributes and        events; forming object-oriented labeled datasets based on the        event and the attributes of each of the datasets; forming a        library or framework of classes with a plurality of        object-orientation modelors; and forming a plurality of        object-manipulation functions, each function being configured to        leverage a respective class among the library or framework of        classes; receiving, with one or more processors, a request to        determine a set of actions to achieve, or increase likelihood        of, a given targeted action; assigning, with one or more        processors, the given targeted action to a first subset of        classes from the library or framework of classes of the        object-orientated application or service; and determining, with        one or more processors, based on the assigning, the set of        actions to achieve, or increase likelihood of, the given        targeted action using a first subset of the plurality of        object-manipulation functions leveraging the first subset of        classes from the library or framework of classes of the        object-orientated application or service.    -   2A. The medium of embodiment 1A, wherein the orchestrating        further comprises: adding version numbers to the datasets;        adding primary surrogate keys to the datasets and updating the        version numbers; and encoding the datasets in dimensional star        schema and updating the version numbers.    -   3A. The medium of any one of embodiments 1A-2A, wherein the        orchestrating further comprises: forming a first training        dataset from the datasets; training, with one or more        processors, a first machine-learning model on the first training        dataset by adjusting parameters of the first machine-learning        model to optimize a first objective function that indicates an        accuracy of the plurality of object-orientation modelors in        generating the library or framework of classes; and storing,        with one or more processors, the adjusted parameters of the        trained first machine-learning model in memory.    -   4A. The medium of embodiment 3A, wherein training comprises        training with gradient descent.    -   5A. The medium of any one of embodiments 1A-4A, wherein the        orchestrating further comprises: forming a second training        dataset from the datasets; training, with one or more        processors, a second machine-learning model on the second        training dataset by adjusting parameters of the second        machine-learning model to optimize a second objective function        that determines the first subset of the plurality of        object-manipulation functions; and storing, with one or more        processors, the adjusted parameters of the trained second        machine-learning model in memory.    -   6A. The medium of any one of embodiments 1A-5A, wherein the        plurality of entity logs comprise: consumers; communications to        consumers by an enterprise; communications to an enterprise by        consumers; purchases by consumers from an enterprise;        non-purchase interactions by consumers with an enterprise; and a        customer relationship management system of an enterprise.    -   7A. The medium of embodiment 6A, wherein: the enterprise is a        credit card issuer and the given targeted action is predicting        whether a consumer will default; the enterprise is a lender and        the given targeted action is predicting whether a consumer will        borrow; the enterprise is an insurance company and the given        targeted action is predicting whether a consumer will file a        claim; the enterprise is an insurance company and the given        targeted action is predicting whether a consumer will sign-up        for insurance; the enterprise is a vehicle seller and the given        targeted action is predicting whether a consumer will purchase a        vehicle; the enterprise is a seller of goods and the given        targeted action is predicting whether a consumer will file a        warranty claim, the enterprise is a wireless operator and the        given targeted action is predicting whether a consumer upgrade        their cellphone, or the enterprise is a bank and the given        targeted action is predicting GDP variation.    -   8A. The medium of any one of embodiments 1A-7A, wherein the        assignment of the given targeted action to a first subset of        classes comprises: assigning the given targeted action to a        first subset of the plurality of objects using a second subset        of the plurality of object-manipulation functions; and        determining the first subset of classes from the library or        framework of classes of the object-orientated application or        service that are related to the first subset of the plurality of        objects.    -   9A. The medium of embodiment 8A, wherein second subset of the        plurality of object-manipulation functions are configured to add        new objects to the plurality of objects.    -   10A. The medium of embodiment 9A, wherein the new objects        comprise attributes and events related to the given targeted        action.    -   11A. The medium of any one of embodiments 1A-10A, wherein the        datasets comprise: a data frame; a data stream; a column in a        table; a row in a column; a cell in a table; structured data;        and unstructured data.    -   12A. The medium of any one of embodiments 1A-11A, wherein the        plurality of object-manipulation functions comprises: a sequence        function used to change a collection of events into a time        sequences for processing; a feature function used to gather        features of a first object-orientation modelor and then use the        features in a second object-orientation modelor; an economic        function used to: gather economic objectives and economic        constraints of an entity; and employ an allocation algorithm to        maximize the objectives; and an ensembling function used to        combine a first subset of the library or framework of classes.    -   13A. The medium of embodiment 12A, wherein the plurality of        object-manipulation functions are arranged to perform in series.    -   14A. The medium of embodiment 12A, wherein the plurality of        object-manipulation functions are arranged to change orders        dynamically based on the given targeted action.    -   15A. The medium of any one of embodiments 1A-14A, wherein the        plurality of object-orientation modelors comprises: a scaled        propensity modelor used to calculate probability of a customer        making an economic commitment; a timing modelor used to        calibrate moments in time when a customer is likely to engage        with the given targeted action; an affinity modelor used to        capture ranked likes and dislikes of an entity's customers for a        first subset of targeted actions; a best action modelor used to        create a framework for concurrent Key Performance Index of the        given targeted action at different points in a customer's        journey; and a cluster modelor used to group an entity's        customers based on the customers' behavior into a finite list.    -   16A. The medium of embodiment 15A, wherein the plurality of        object-orientation modelors are arranged to perform in series or        in parallel.    -   17A. The medium of embodiment 15A, wherein the plurality of        object-orientation modelors are arranged to change orders        dynamically based on the given targeted action.    -   18A. The medium of any one of embodiments 1A-17A, wherein: the        given targeted action comprises a plurality of sub-targets; and        at least some targets of the plurality of sub-targets are        expected to happen at different times in future.    -   19A. The medium of any one of embodiments 1A-18A, wherein: the        given targeted action comprises a plurality of sub-targets; and        the plurality of object-orientation modelors comprises: a scaled        propensity modelor used to calculate probability of a customer        making an economic commitment; a timing modelor used to        calibrate moments in time when a customer is likely to engage        with each subset of the plurality of sub-targets; an affinity        modelor used to capture ranked likes and dislikes of an entity's        customers for a first subset of targeted actions; a best action        modelor used to create a framework for concurrent Key        Performance Index for each subset of the plurality of        sub-targets at different points in a customer's journey; a        cluster modelor used to group an entity's customers based on the        customers' behavior into a finite list; and wherein: a first        subset of the plurality of object-orientation modelors are used        for a first subset of the plurality of sub-targets; a second        subset of the plurality of object-orientation modelors are used        for a second subset of the plurality of sub-targets; and wherein        the order in which the first subset of the plurality of        object-orientation modelors perform is different from the order        in which the second subset of the plurality of        object-orientation modelors perform.    -   20A. The medium of any one of embodiments 1A-10A, wherein the        object-oriented labeled datasets formed according to an ontology        of events.    -   21A. The medium of embodiment 20A, wherein the ontology of        events comprises Concurrent Ontology Labelling Datastore (COLD)        methodology.    -   22A. The medium of any one of embodiments 1A-21A, wherein the        object-oriented labeled datasets formed according to a        hierarchal taxonomy of events.    -   1B. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: identifying, with one or more        processors, feature processing transformations of one or more        datasets; identifying, with one or more processors, pipelines of        the transformations, the pipelines being pipelines in an        object-oriented modeling application; forming, with one or more        processors, a first plurality of classes using object-oriented        modeling of the feature processing transformations of the one or        more datasets, the first plurality of classes being classes of        objects in the object-oriented modeling application; forming,        with one or more processors, a second plurality of classes using        object-oriented modeling of the pipelines, the second plurality        of classes being classes of objects in the object-oriented        modeling application; forming, with one or more processors, a        third plurality of classes using object-oriented modeling of the        one or more datasets, the third plurality of classes being        classes of objects in the object-oriented modeling application;        interrogating, with one or more processors, a class library        containing the first plurality of classes to determine first        class definition information; interrogating, with one or more        processors, a class library containing the second plurality of        classes to determine second class definition information;        selecting, with one or more processors, a given dataset from the        one or more datasets or other datasets; interrogating, with one        or more processors, a class library containing the third        plurality of classes to determine third class definition        information; accessing, with one or more processors, the first,        second, and third class definitions information to produce an        interdependency graph of one or more data processing operator        instances of a given pipeline among the pipelines of the        transformations; generating, with one or more processors, an        execution schedule of the given pipeline based on the        interdependency graph; causing, with one or more processors,        execution of the given pipeline according to the execution        schedule; accessing the first definition information to process        the given dataset; and storing a result of processing the given        dataset in memory.    -   2B. The medium of embodiment 1B, wherein the operations further        comprise: accessing attributes of the given dataset at each of a        plurality of modelors of the given pipeline.    -   3B. The medium of any one of embodiments 1B-2B, wherein causing        execution of the given pipeline according to the execution        schedule comprises: executing the given pipeline on a        distributed cluster computing framework.    -   4B. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: writing, with one or more        processors, a first plurality of classes using object-oriented        modelling; writing, with one or more processors, a second        plurality of classes using object-oriented modelling of modeling        topics; scanning, with one or more processors, a class library        containing the first plurality of classes to determine class        definition information; scanning, with one or more processors, a        class library containing the second plurality of classes to        determine class definition information; receiving, with one or        more processors, at an orchestrating system, from a subscribing        modeling object a request for a subscription to a given modeling        topic in a given modeling topic class among the second plurality        of classes, the subscription request including a modeling topic        filter to select the given modeling topic from a plurality of        modeling topics described by the given modeling topic class;        registering, with one or more processors, by the orchestrating        system, a modeling topic accessor associated with the        subscribing modeling object; registering, with one or more        processors, by the orchestrating system, a modeling topic        mutator associated with the subscribing modeling object;        processing, with one or more processors, by the orchestrating        system, through the modeling topic filter a modeling topic that        is accessed through an accessor and is described by the modeling        topic class, the modeling topic being received from a modeling        publisher object; notifying, with one or more processors, by the        orchestrating system, the subscribing object of the received        modeling topic through a registered modeling topic listener, in        response to determining that the received modeling topic matches        the modeling topic filter; and mutating, with one or more        processors, the received modeling topic at the subscriber        modeling object in response to determining that the received        modeling topic matches the modeling topic filter included in the        request for a subscription.    -   5B. The medium of embodiment 4B, wherein mutating comprises:        adding an attribute to an object, deleting an attribute of an        object, updating an attribute of an object, reading an attribute        an object, adding reference to another object as an attribute,        using a setter, or using a getter.    -   6B. The medium of any one of embodiments 4B-5B, wherein mutating        comprises: adding an attribute to an object, deleting an        attribute of an object, updating an attribute of an object,        reading an attribute an object, adding reference to another        object as an attribute, using a setter, and using a getter.    -   7B. The medium of any one of embodiments 4B-6B, wherein: items        captured in modeling topics include: consumers, communications        to consumers by an enterprise, communications to an enterprise        by consumers, events that include purchases by consumers from        the enterprise, and events that include non-purchase        interactions by consumers with the enterprise; and at least some        items are obtained from entity logs that are obtained from a        customer relationship management system of the enterprise.    -   8B. The medium of embodiment 7B, wherein: a result of the        operations is used by a trained predictive machine learning        model developed using an object-oriented modeling (OOM)        framework.    -   9B. The medium of embodiment 8B, wherein: the enterprise is a        credit card issuer and the trained predictive machine learning        model developed using the OOM framework is configured to predict        whether a consumer will default; the enterprise is a lender and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        will borrow; the enterprise is an insurance company and the        trained predictive machine learning model developed using the        OOM framework is configured to predict whether a consumer will        file a claim; the enterprise is an insurance company and the        trained predictive machine learning model developed using the        OOM framework is configured to predict whether a consumer will        sign-up for insurance; the enterprise is a vehicle seller and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        will purchase a vehicle; the enterprise is a seller of goods and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        will file a warranty claim, or the enterprise is a wireless        operator and the trained predictive machine learning model        developed using the OOM framework is configured to predict        whether a consumer upgrade their cellphone, or the enterprise is        bank and the trained predictive machine learning model developed        using the OOM framework is configured to predict the change in        GDP.    -   10B. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: obtaining, with one or more        processors, for a plurality of entities, datasets, wherein: the        datasets comprise a plurality of entity logs; a first subset of        the plurality of entity logs are events involving the entities;        a first subset of the events are actions by the entities; at        least some of the actions are targeted actions; a second subset        of the plurality of entity logs are attributes related to the        entities; and the events are distinct from the attributes;        forming, with one or more processors, an object-orientated        orchestration, the object-orientated orchestration comprising:        forming a plurality of objects, wherein each object of the        plurality of objects comprises a different set of attributes and        events; forming object-oriented labeled datasets based on the        event and the attributes of each of the datasets; forming a        library of classes, generated by a plurality of        object-orientation modelors; and forming a plurality of        object-manipulation functions, each function configured to        leverage a specific class; forming a first training dataset from        the datasets; training, with one or more processors, a first        machine-learning model on the first training dataset by        adjusting parameters of the first machine-learning model to        optimize a first objective function that indicates        interdependency of the plurality of object-manipulation        functions in leveraging a specific class; forming an        interdependency graph using, at least in part, the first        objective function; and storing, with one or more processors,        the adjusted parameters of the trained first machine-learning        model in memory; receiving a request to determine a set of        actions required to achieve a specific targeted action;        determining, with one or more processors, the set of actions        required to achieve the specific targeted action using a        compiler function, the compiler function comprising: assigning        the specific targeted action to a first subset of the plurality        of objects using a first subset of the plurality of        object-manipulation functions, wherein: the first subset of the        plurality of object-manipulation functions is formed using the        interdependency graph; and determining a first subset of classes        from the library of classes of the object-oriented orchestration        that are related to the first subset of the plurality of        objects; determining the set of actions required to achieve the        specific targeted action using a second subset of the plurality        of object-manipulation functions, wherein: the second subset of        the plurality of object-manipulation functions is formed using        the interdependency graph; and each object-manipulation function        of the second subset of the plurality of object-manipulation        functions is configured to leverage at least one class of the        first subset of classes from the library of classes of the        object-oriented orchestration.    -   11B. The medium of embodiment 10B, wherein the interdependency        graph comprises: a plurality of execution schedules, wherein        each execution schedule from the plurality of execution        schedules comprises a subset of the object-manipulation        functions.    -   12B. The medium of embodiment 11B, wherein each execution        schedule from the plurality of execution schedules is configured        to leverage at least one class from the library of classes.    -   13B. The medium of any one of embodiments 10B-12B, wherein the        interdependency graph comprises: a plurality of execution        triggers, wherein each execution trigger from the plurality of        execution triggers comprises a subset of the object-manipulation        functions.    -   14B. The medium of any one of embodiments 10B-13B, wherein the        interdependency graph comprises: a plurality of execution        schedules, wherein each execution schedule from the plurality of        execution schedules comprises a subset of the        object-manipulation functions, a plurality of execution        triggers, wherein each execution trigger from the plurality of        execution triggers comprises a subset of the object-manipulation        functions, and an orchestrator assigning execution triggers or        execution schedules within the interdependency graph.    -   15B. The medium of any one of embodiments 10B-14B, wherein the        formation object-orientated orchestration further comprises:        forming a second training dataset from the datasets; training,        with one or more processors, a second machine-learning model on        the second training dataset by adjusting parameters of the        second machine-learning model to optimize a second objective        function that determines the first subset of the plurality of        object-manipulation functions; and storing, with one or more        processors, the adjusted parameters of the trained second        machine-learning model in memory.    -   16B. The medium of any one of embodiments 10B-15B, comprising        indicating interdependency of the plurality of        object-manipulation functions in leveraging a specific class        with the first trained machine learning at least in part by:        obtaining a given entity log of the given entity; determining a        plurality of features from the given entity log, the plurality        of features having fewer dimensions than the given entity log;        and inputting the determined plurality of features into the        first trained machine learning model to cause the model to        output a value indicative of the interdependency of the        plurality of object-manipulation functions in leveraging a        specific class related to the given entity.    -   17B. The medium of any one of embodiments 10B-16B, wherein: the        first machine learning model is based on a plurality of decision        trees combined with an ensemble procedure; and the ensemble        procedure is boosting, random forest or other form of bootstrap        aggregation, or rotation forest; and at least some of the        decision trees are trained with classification and regression        tree by recursively splitting a feature space of inputs to the        first machine learning model along different dimensions of the        feature space at values of respective dimensions that locally        optimize the respective split to minimize entropy of Gini        impurity of targeted actions and non-targeted actions on each        side of respective splits.    -   18B. The medium of any one of embodiments 10B-17B, comprising,        before training, transforming each entity log into a collection        of features to which the first machine learning model is capable        of responding and training the model on features of the        collection of features.    -   19B. The medium of embodiment 18, wherein at least some of the        features are determined by the attributes, the attributes        comprising: entity restrictions for at least some of the        plurality of entities; entity business protocols for at least        some of the plurality of entities; entity policies for at least        some of the plurality of entities; entity authorized users for        at least some of the plurality of entities; and entity security        protocols for at least some of the plurality of entities.    -   20B. The medium of any one of embodiments 10B-19B, wherein        training comprises means for training.    -   21B. The medium of any one of embodiments 10B-20B, wherein the        plurality of entity logs comprise information about: consumers;        communications to consumers by an enterprise; communications to        an enterprise by consumers; purchases by consumers from an        enterprise; non-purchase interactions by consumers with an        enterprise; and a customer relationship management system of an        enterprise.    -   22B. The medium of any one of embodiments 10B-21B, wherein: the        enterprise is a credit card issuer and the specific targeted        action is predicting whether a consumer will default; the        enterprise is a lender and the specific targeted action is        predicting whether a consumer will borrow; the enterprise is an        insurance company and the specific targeted action is predicting        whether a consumer will file a claim; the enterprise is an        insurance company and the specific targeted action is predicting        whether a consumer will sign-up for insurance; the enterprise is        a vehicle seller and the specific targeted action is predicting        whether a consumer will purchase a vehicle; the enterprise is a        seller of goods and the specific targeted action is predicting        whether a consumer will file a warranty claim, the enterprise is        a wireless operator and the specific targeted action is        predicting whether a consumer upgrade their cellphone, or the        enterprise is a bank and the specific targeted action is        predicting GDP variation.    -   23B. The medium of any one of embodiments 10B-22B, wherein the        first trained model is configured to filter some of the entity        logs, wherein the filtration comprise: a dependency level among        the entity logs calculated by Bayesian Networks; a logistic        regression calculated by Lasso and ElasticNet penalty functions;        or a product moment correlation coefficient calculated by        Pearson correlation.    -   24B. The medium of any one of embodiments 10B-23B, wherein the        object-oriented labeled datasets formed according to an ontology        of events.    -   25B. The medium of any one of embodiments 10B-24B, wherein the        object-oriented labeled datasets formed according to a        hierarchal taxonomy of events.    -   26B. The medium of any one of embodiments 10B-24B, the        operations further comprising: steps for determining the set of        actions required to achieve the specific targeted action.    -   27B. The medium of any one of embodiments 10B-26B, wherein the        operations further comprise: formation of the interdependency        graph using, at least in part, feature engineering modelors, the        feature engineering modelors comprise: recency feature        engineering modelors; frequency feature engineering modelors;        lag feature engineering modelors; difference feature engineering        modelors; and harmonic analysis feature engineering modelors;        wherein the feature engineering modelors are a subset of        plurality of object-orientation modelors.    -   28B. The medium of any one of embodiments 10B-27B, wherein the        plurality of object-manipulation functions comprise: a feature        engineering function used to gather features of a first        object-orientation modelor and then use the features in a second        object-orientation modelor, wherein the feature engineering        function comprises: a recency feature engineering sub-routine; a        frequency feature engineering sub-routine; a lag feature        engineering sub-routine; a difference feature engineering        sub-routine; and a harmonic analysis feature engineering        sub-routine.    -   29B. The medium of any one of embodiments 10B-28B, wherein the        plurality of object-orientation modelors comprise: a scaled        propensity modelor used to calculate probability of a customer        making an economic commitment; a timing modelor used to        calibrate moments in time when a customer is likely to engage        with the specific targeted action; an affinity modelor used to        capture ranked likes and dislikes of an entity's customers for a        first subset of targeted actions; a best action modelor used to        create a framework for concurrent Key Performance Index of the        specific targeted action at different points in a customer's        journey; and a cluster modelor used to group an entity's        customers based on the customers' behavior into a finite list.

30B. The medium of any one of embodiments 10B-29B, the operationsfurther comprising forming an object-orientated orchestration by: addingversion indicators to the datasets; adding primary surrogate keys to thedatasets and updating the version indicators; and encoding the datasetsin dimensional star schema and updating the version indicators.

-   -   31B. The medium of any one of embodiments 10B-30B, the        operations further comprising forming an object-orientated        orchestration by: adding version indicators to the modelors.    -   1C. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: forming, with one or more        processors, a first plurality of classes using object-oriented        modelling of modelling objects, the modelling objects being used        to implement a machine learning design based on optimization        criteria using an object-oriented modeling (OOM) framework, the        first plurality of classes being members of a first class        library; forming, with one or more processors, a second        plurality of classes using object-oriented modelling of an        orchestration of the machine learning design, the second        plurality of classes being members of a second class library;        forming, with one or more processors, a third plurality of        classes using object-oriented modelling of the optimization        criteria, the optimization criteria being used to optimize an        orchestration of the modelling objects, the third plurality of        classes being members of a third class library; forming, with        one or more processors, a fourth plurality of classes using        object-oriented modelling of the optimization of the        orchestration of the modelling objects, the fourth plurality of        classes being members of a fourth class library; forming, with        one or more processors, a fifth plurality using object-oriented        modelling of an optimization value, the fifth plurality of        classes being members of a fifth class library; accessing, with        one or more processors, an optimization criterion object from        the third class library; accessing, with one or more processors,        the third class library to determine first class definition        information, the first class definition information being class        definition information of the optimization criterion object;        accessing, with one or more processors, the first class library        to determine second class definition information; accessing,        with one or more processors, the fourth class to determine third        class definition information; using, with one or more        processors, at least some of the first, second, or third class        definition information to form a sequence of object manipulation        function to effectuate access by an orchestration system to        methods and attributes of at least some of the first, second,        third, fourth, or fifth plurality of classes to manipulate        objects of the first plurality of classes; using, with one or        more processors, at least some of the first, second, or third        class definition information to effectuate access to the object        manipulation functions by the orchestration system; processing,        with one or more processors, orchestration of modelling, the        orchestration including statements seeking access to one or more        modelling object classes among the first plurality of classes        within the first class library; processing, with one or more        processors, an optimization method of orchestration, the        optimization method including statements seeking access to one        or more orchestration object classes among the second plurality        of classes within the second class library; processing, with one        or more processors, optimization criteria of the optimization        method, the optimization criteria including statements seeking        access to one or more orchestration object classes among the        second plurality of classes within the second class library; and        invoking, with one or more processors, previously formed object        manipulation functions using at least some of the sought access        to activate the object manipulation functions during        implementation by the optimization system of the optimization        method, thereby causing use of the optimization to access        modelling object classes among the first plurality of classes.    -   2C. The medium of embodiment 1C, wherein the instructions are        executed in the optimization system to execute optimization the        orchestration of modelling objects for implementation of the        machine learning design based on optimization criteria using an        object-oriented modeling (OOM) framework.    -   3C. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system to effectuate operations to execute the        processing of modeling methods organized in racks of a machine        learning pipeline to facilitate optimization of performance        using modelling methods for implementation of machine learning        design in an object-oriented modeling (OOM) framework, the        operations comprising: writing, with the computer system, a        first plurality of classes of a first class library using        object-oriented modelling of optimization methods; writing, with        the computer system, a second plurality of classes of a second        class library using object-oriented modelling of said modelling        methods; writing, with the computer system, parameters and        hyper-parameters of the modeling methods as attributes as the        modeling methods; writing, with the computer system, a plurality        of classes of a third class library using object-oriented        modelling of the modelling racks; scanning, with the computer        system, modelling racks classes among the third plurality of        classes in the third class library to determine first class        definition information; selecting, with the computer system, a        collection of one or more racks among the racks of the machine        learning pipeline; for each rack in the collection, with the        computer system, selecting one or more modeling method objects;        scanning, with the computer system, modelling method classes        among the second plurality of classes in the second class        library to determine second class definition information;        assigning, with the computer system, one or more racks and        locations within the one or more racks to corresponding selected        modeling method objects; invoking, with the computer system, at        least some of the second class definition information of        modeling method classes and at least some of the first class        definition information of the modeling racks to produce object        manipulation functions that allow the computer system to access        the methods and attributes of at least some of the modeling        method objects, at least some of the manipulation functions        being configured to effectuate writing locations within racks        and attributes of racks; selecting, with the computer system, an        optimization method among the optimization methods; and        creating, with the computer system, an optimization object.    -   4C. The medium of embodiment 3C, the operations further        comprising: invoking the class definition information of at        least one modeling method class and at least some class        definition information of at least one modeling rack and at        least some class definition information of at least one        optimization method to produce additional object manipulation        functions that allow the computing system to access methods and        attributes of a corresponding optimization object to manipulate        the corresponding optimization object; and invoking the class        definition of the least one optimization method to optimize the        corresponding optimization object.    -   5C. The medium of any one of embodiments 3C-4C, wherein: the        operations further comprise processing data construct objects        based on entity logs; entities captured in the data construct        objects comprise: consumers, communications to consumers by an        enterprise, communications to an enterprise by consumers, and        events that include purchases by consumers from the enterprise        and non-purchase interactions by consumers with the enterprise;        and the entity logs are obtained from a customer relationship        management system of the enterprise.    -   6C. The medium of embodiment 5C, wherein: the enterprise is a        credit card issuer and a trained predictive machine learning        model developed using the object-oriented modeling (OOM)        framework is configured to predict whether a consumer will        default; the enterprise is a lender and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will borrow; the        enterprise is an insurance company and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will file a claim; the        enterprise is an insurance company and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will sign-up for        insurance; the enterprise is a vehicle seller and the trained        predictive machine learning model developed using the OOM        framework is configured to predict whether a consumer will        purchase a vehicle; the enterprise is a seller of goods and the        trained predictive machine learning model developed using the        OOM framework is configured to predict whether a consumer will        file a warranty claim; the enterprise is a wireless operator and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        upgrade their cellphone; or the enterprise is bank and the        trained predictive machine learning model developed using the        OOM framework is configured to predict the change in GDP.    -   7C. The medium of any one of embodiments 3C-6C, wherein: the        operations further comprise processing data construct objects        based on entity logs; entities captured in the data construct        objects comprise: consumers, product information, service        information, events that include purchases by consumers from the        enterprise and non-purchase interactions by consumers with the        enterprise, and events that include subscriptions by consumers        from the enterprise and non-purchase interactions by consumers        with the enterprise; and the entity logs are obtained from a        customer relationship management system of the enterprise.    -   8C. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: obtaining, with one or more        processors, for a plurality of entities, datasets, wherein: the        datasets comprise a plurality of entity logs; a first subset of        the plurality of entity logs are events involving the entities;        a first subset of the events are actions by the entities; at        least some of the actions are targeted actions; a second subset        of the plurality of entity logs are attributes related to the        entities; and the events are distinct from the attributes;        forming, with one or more processors, an object-orientated        orchestration by: forming a plurality of objects, wherein each        object of the plurality of objects comprises a different set of        attributes and events; forming object-oriented labeled datasets        based on the event and the attributes of each of the datasets;        forming a library of classes, generated by a plurality of        object-orientation modelors; and forming a plurality of        object-manipulation functions, each function configured to        leverage a specific class; receiving, with one or more        processors, a request to determine a set of actions required to        achieve a specific targeted action; and determining, with one or        more processors, the set of actions required to achieve the        specific targeted action using a compiler function, the compiler        function comprising instructions to effectuate: assigning the        specific targeted action to a first subset of the plurality of        objects using a first subset of the plurality of        object-manipulation functions; and determining a first subset of        classes from the library of classes of the object-oriented        orchestration that are related to the first subset of the        plurality of objects; determining the set of actions required to        achieve the specific targeted action using a second subset of        the plurality of object-manipulation functions, wherein: each        object-manipulation function of the second subset of the        plurality of object-manipulation functions is configured to        leverage at least one class of the first subset of classes from        the library of classes of the object-oriented orchestration.    -   9C. The medium of embodiment 8C, wherein the compiler function        further comprising instructions to effectuate: forming a first        training dataset from the datasets; training a first        machine-learning model on the first training dataset by        adjusting parameters of the first machine-learning model to        optimize a first objective function that indicates which        object-manipulation functions from the plurality of        object-manipulation functions should be included in the second        subset of the plurality of object-manipulation functions; and        storing the adjusted parameters of the trained first        machine-learning model in memory.    -   10C. The medium of embodiment 9C, wherein: the first machine        learning model comprises a Hidden Markov model.    -   11C. The medium of embodiment 9C, wherein: the first machine        learning model comprises a long short-term memory model.    -   12C. The medium of embodiment 9C, wherein: the first machine        learning model comprises a dynamic Bayesian network.    -   13C. The medium of embodiment 9C, wherein: the first machine        learning model comprises a neural network classifier.    -   14C. The medium of embodiment 9C, wherein: the first machine        learning model is part of a value function or an environment        model of a reinforcement learning model.    -   15C. The medium of embodiment 9C, wherein training comprises        steps for training.    -   16C. The medium of embodiment 9C, wherein the first dataset        comprises: the first subset of the plurality of objects.    -   17C. The medium of embodiment 9C, comprising: inputting more        than 1,000 entity logs corresponding to more than 1,000 entities        input into the first trained machine learning model.    -   18C. The medium of any one of embodiments 8C-17C, wherein the        compiler function further comprising instructions to effectuate:        forming a first training dataset from the datasets; training a        first machine-learning model on the first training dataset by        adjusting parameters of the first machine-learning model to        optimize a first objective function that indicates which        object-manipulation functions from the plurality of        object-manipulation functions should be included in the second        subset of the plurality of object-manipulation functions; and        storing the adjusted parameters of the trained first        machine-learning modelling pipeline in memory.    -   19C. The medium of any one of embodiments 8C-18C, wherein the        compiler function further comprising instructions to effectuate:        forming a second training dataset from the datasets; training a        second machine-learning model on the second training dataset by        adjusting parameters of the second machine-learning model to        optimize a second objective function that indicates the order of        operation of each object-manipulation function in the second        subset of the plurality of object-manipulation functions; and        storing the adjusted parameters of the trained second        machine-learning model in memory.    -   20C. The medium of any one of embodiments 8C-19C, wherein the        plurality of entity logs comprise information about: consumers;        communications to consumers by an enterprise; communications to        an enterprise by consumers; purchases by consumers from an        enterprise; non-purchase interactions by consumers with an        enterprise; and a customer relationship management system of an        enterprise.    -   21C. The medium of embodiment 20C, wherein: the enterprise is a        credit card issuer and the specific targeted action is        predicting whether a consumer will default; the enterprise is a        lender and the specific targeted action is predicting whether a        consumer will borrow; the enterprise is an insurance company and        the specific targeted action is predicting whether a consumer        will file a claim; the enterprise is an insurance company and        the specific targeted action is predicting whether a consumer        will sign-up for insurance; the enterprise is a vehicle seller        and the specific targeted action is predicting whether a        consumer will purchase a vehicle; the enterprise is a seller of        goods and the specific targeted action is predicting whether a        consumer will file a warranty claim, the enterprise is a        wireless operator and the specific targeted action is predicting        whether a consumer upgrade their cellphone, or the enterprise is        a bank and the specific targeted action is predicting GDP        variation.    -   22C. The medium of embodiment 20C, wherein the second subset of        the plurality of object-manipulation functions further        comprises: an economic optimization function with a plurality of        function parameters, wherein the plurality of function        parameters are adjusted based on business objectives of the        enterprise.    -   23C. The medium of embodiment 22C, wherein the business        objectives of the enterprise comprise: increase in revenue;        increase in profit margin; or reduction in cost; wherein each of        the business objectives has a set business constraints.    -   24C. The medium of embodiment 23C, wherein the at least some of        the sets business constraints comprise: a plurality of specific        age groups; a specific product or service; a window of time; and        a plurality of geographical locations.    -   25C. The medium of embodiment 23C, wherein the at least some of        the sets business constraints comprise: an amount of time to        optimize operation; and an amount of hardware used to optimize        operation.    -   26C. The medium of any one of embodiments 8C-25C, wherein: the        specific targeted comprises a plurality of sub-targets; and the        plurality of object-orientation modelors comprises: a scaled        propensity modelor used to calculate probability of a customer        making an economic commitment; a timing modelor used to        calibrate moments in time when a customer is likely to engage        with each subset of the plurality of sub-targets; an affinity        modelor used to capture ranked likes and dislikes of an entity's        customers for a first subset of targeted actions; a best action        modelor used to create a framework for concurrent Key        Performance Index for each subset of the plurality of        sub-targets at different points in a customer's journey; a        cluster modelor used to group an entity's customers based on the        customers' behavior into a finite list; and wherein: a first        subset of the plurality of object-orientation modelors are used        for a first subset of the plurality of sub-targets; a second        subset of the plurality of object-orientation modelors are used        for a second subset of the plurality of sub-targets; and wherein        the order in which the first subset of the plurality of        object-orientation modelors perform is different from the order        in which the second subset of the plurality of        object-orientation modelors perform.    -   1D. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: writing, with a computing        system, a first plurality of classes using object-oriented        modelling of modelling methods; writing, with the computing        system, a second plurality of classes using object-oriented        modelling of governance; scanning, with the computing system, a        set of libraries collectively containing both modelling object        classes among the first plurality of classes and governance        classes among the second plurality of classes to determine class        definition information; using, with the computing system, at        least some of the class definition information to produce object        manipulation functions, wherein the object manipulation        functions allow a governance system to access methods and        attributes of classes among first plurality of classes or the        second plurality of classes to manipulate objects of at least        some of the modelling object classes; and using at least some of        the class definition information to effectuate access to the        object manipulation functions.    -   2D. The medium of embodiment 1D, wherein: the operations execute        quality management of modelling methods for implementation of        machine learning design in an object-oriented modeling (OOM)        framework.    -   3D. The medium of any one of embodiments 1D-2D, wherein: the        modeled governance comprises a set of structures, processes, or        policies by which pipeline development, deployment, or use        functionality within an organization or set of organizations is        directed, managed, or controlled.    -   4D. The medium of any one of embodiments 1D-3D, wherein: the        modeled governance comprises a set of structures, processes, and        policies by which pipeline development, deployment, and use        functionality within an organization or set of organizations is        directed, managed, and controlled.    -   5D. The medium of any one of embodiments 1D-4D, wherein: the        modeled governance comprises a policy; and the policy comprises        a set of rules, controls, or resolutions put in place to dictate        model behavior.    -   6D. The medium of any one of embodiments 1D-5D, wherein: the        modeled governance comprises a policy; and the policy comprises        a set of rules, controls, or resolutions put in place to dictate        model versioning.    -   7D. The medium of any one of embodiments 1D-6D, wherein: the        modeled governance comprises a set of policors; and        meta-policies having detection rules are used to detect        conflicts among policors.    -   8D. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system effectuate operations to manage compliance        governance using modelling methods in a pipeline for        implementation of machine learning design in an object-oriented        modeling (OOM) framework, the operations comprising: forming,        with one or more processors, a first plurality of classes using        object-oriented modelling of the modelling methods; forming,        with one or more processors, a second plurality of classes using        object-oriented modelling of governance compliance methods;        scanning, with one or more processors, a class library        containing modelling method classes to determine a first part of        class definition information; scanning, with one or more        processors, a class library containing governance compliance        classes to determine a second part of class definition        information; and using, with one or more processors, the first        part of class definition information of the modeling method        class and the second part of class definition information of the        management method class to produce object manipulation functions        that allow the computing system to access methods and attributes        of a governance compliance object to manipulate a method class        object.    -   9D. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system effectuate operations to measure quality of        processing of data constructs in a pipeline using modelling        methods for implementation of machine learning design in an        object-oriented modeling (OOM) framework, the operations        comprising: writing, with the computing system, a first        plurality of classes using object-oriented modelling of the        modelling methods; writing, with the computing system, a second        plurality of classes using object-oriented modelling of quality        measurement methods; scanning, with the computing system, a        class library containing modelling method classes to determine a        first part of class definition information; scanning, with the        computing system, another class library containing quality        management classes to determine a second part of class        definition information; and invoking, with the computing system,        the class definition information to produce object manipulation        functions that allow the computing system to access methods and        attributes of data classes to manipulate a modeling method        class.    -   10D. The medium of embodiment 9D, wherein: the quality        measurement methods comprise data quality monitoring (DQM),        model quality monitoring (MQM), score quality monitoring (SQM),        bias quality management (BQM), privacy quality management (PQM),        or label quality monitoring (LQM).    -   11D. The medium of any one of embodiments 9D-10D, wherein: the        operations comprise object manipulation by allowing reading of        attributes, usage of a given modeling method, audit of usage of        a given modeling object, reporting attempts to use the given        modeling object, or verifying proper licensing.    -   12D. The medium of any one of embodiments 9D-11D, wherein: the        operations comprise object manipulation by allowing reading of        attributes, usage of a given modeling method, audit of usage of        a given modeling object, reporting attempts to use the given        modeling object, and verifying proper licensing; and the quality        measurement methods comprise data quality monitoring (DQM),        model quality monitoring (MQM), score quality monitoring (SQM),        bias quality management (BQM), privacy quality management (PQM),        and label quality monitoring (LQM).    -   13D. The medium any one of embodiments 9D-12D, wherein: the        operations further comprise processing data construct objects        based on entity logs; entities captured in the data construct        objects comprise: consumers, communications to consumers by an        enterprise, communications to an enterprise by consumers, and        events that include purchases by consumers from the enterprise        and non-purchase interactions by consumers with the enterprise;        and the entity logs are obtained from a customer relationship        management system of the enterprise.    -   14D. The medium any one of embodiments 9D-13D, wherein: the        enterprise is a credit card issuer and a trained predictive        machine learning model developed using the object-oriented        modeling (OOM) framework is configured to predict whether a        consumer will default; the enterprise is a lender and the        trained predictive machine learning model developed using the        OOM framework is configured to predict whether a consumer will        borrow; the enterprise is an insurance company and the trained        predictive machine learning model developed using the OOM        framework is configured to predict whether a consumer will file        a claim; the enterprise is an insurance company and the trained        predictive machine learning model developed using the OOM        framework is configured to predict whether a consumer will        sign-up for insurance; the enterprise is a vehicle seller and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        will purchase a vehicle; the enterprise is a seller of goods and        the trained predictive machine learning model developed using        the OOM framework is configured to predict whether a consumer        will file a warranty claim; the enterprise is a wireless        operator and the trained predictive machine learning model        developed using the OOM framework is configured to predict        whether a consumer upgrade their cellphone; or the enterprise is        bank and the trained predictive machine learning model developed        using the OOM framework is configured to predict the change in        GDP.    -   15D. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors        effectuate operations comprising: obtaining, with one or more        processors, for a plurality of entities, datasets, wherein: the        datasets comprise a plurality of entity logs; a first subset of        the plurality of entity logs are events involving the entities;        a first subset of the events are actions by the entities; at        least some of the actions are targeted actions; a second subset        of the plurality of entity logs are attributes related to the        entities; a first subset of the attributes are governance        attributes; and the events are distinct from the attributes;        forming, with one or more processors, an object-orientated        orchestration, the object-orientated orchestration comprising:        forming a plurality of objects, wherein each object of the        plurality of objects comprises a different set of attributes and        events; forming object-oriented labeled datasets based on the        event and the attributes of each of the datasets; forming a        library of classes, generated by a plurality of        object-orientation modelors; and forming a plurality of        object-manipulation functions, each function configured to        leverage a specific class; receiving a request to determine a        set of actions required to achieve a specific targeted action;        assigning the specific targeted action to a first subset of        classes from the library of classes of the object-oriented        orchestration; and determining, with one or more processors, the        set of actions required to achieve the specific targeted action        using a first subset of the plurality of object-manipulation        functions related to the first subset of classes from the        library of classes of the object-oriented orchestration.    -   16D. The medium of embodiment 15D, wherein the governance        attributes comprises: entity restrictions for at least some of        the plurality of entities; entity business protocols for at        least some of the plurality of entities; entity policies for at        least some of the plurality of entities; entity authorized users        for at least some of the plurality of entities; and entity        security protocols for at least some of the plurality of        entities.    -   17D. The medium of any one of embodiments 15D-16D, wherein: a        first subset of the plurality of object-orientation modelors are        governance modelors; the governance modelors are configured to        form governance classes; and a second subset of the plurality of        object-manipulation functions are governance functions, wherein:        the governance functions are configured to leverage at least one        of the governance classes.    -   18D. The medium of embodiment 17D, wherein: a first subset of        the governance modelors are ontology governance modelors; and a        second subset of the governance modelors are taxonomy governance        modelors.    -   19D. The medium of embodiment 17D, wherein the formation        object-orientated orchestration further comprises: forming a        first training dataset from the datasets; training, with one or        more processors, a first machine-learning model on the first        training dataset by adjusting parameters of the first        machine-learning model to optimize a first objective function        that indicates an accuracy of the governance functions in        complying with the governance attributes; and storing, with one        or more processors, the adjusted parameters of the trained first        machine-learning model in memory.    -   20D. The medium of any one of embodiments 15D-19D, wherein: the        governance attributes comprise a plurality of access levels for        entity users of at least some of the plurality of entities; and        the specific targeted action comprises a plurality of        sub-targets, wherein: each of the plurality of sub-targets is        assigned with a subset of the plurality of access levels.    -   21D. The medium of any one of embodiments 15D-20D, wherein the        plurality of object-manipulation functions comprise: a sequence        function used to change a collection of events into a time        sequences for processing; a feature function used to gather        features of a first object-orientation modelor and then use the        features in a second object-orientation modelor; an economic        function used to: gather economic objectives and economic        constraints of an entity; and employ an allocation algorithm to        maximize the objectives; and an ensembling function used to        combine a first subset of the library of classes.    -   22D. The medium of embodiment 21D, wherein the plurality of        object-manipulation functions are arranged to change orders        dynamically based on the specific targeted action.    -   23D. The medium of any one of embodiments 15D-22D, wherein the        plurality of object-orientation modelors comprise: ingestion        modelors used to control schema drift of the datasets and add        version numbers to the datasets; landing modelors used to clean        error records in the datasets and update the version numbers of        the datasets; curation modelors used to normalize the datasets,        by adding primary surrogate keys, and update the version numbers        of the datasets; dimensional modelors used to encode the        datasets in dimensional star schema and update the version        numbers of the datasets; and feature and label modelors used to:        change the datasets from dimensional star schema to denormalized        flat table; adjust granularity of the datasets; and update the        version numbers of the datasets.    -   24D. The medium of any one of embodiments 15D-23D, wherein the        datasets comprise: training datasets, used to fit parameters of        the object-orientation modelors; validation datasets, used to        tune the parameters of the object-orientation modelors; quality        assurance datasets, used to test accuracy of the        object-orientation modelors; association datasets, used to        relate datasets to each other; and targeted action datasets,        used to determine the set of actions required to achieve the        specific targeted action.    -   25D. The medium of any one of embodiments 15D-24D, wherein: each        action from the set of actions is assigned with a score, the        score indicating impact level of the action in achieving the        specific targeted action.    -   26D. The medium of any one of embodiments 15D-25D, wherein the        formation object-orientated orchestration further comprises:        forming a first training dataset from the datasets; training a        first machine-learning model on the first training dataset by        adjusting parameters of the first machine-learning model to        optimize a first objective function that indicates an accuracy        of the plurality of object-orientation modelors in complying        with the governance attributes; and storing the adjusted        parameters of the trained first machine-learning model in        memory.    -   1E. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system effectuate operations to execute quality        management of modelling methods for implementation of a machine        learning design in an object-oriented modeling (OOM) framework,        the operations comprising: writing, with the computing system,        modelling-object classes using object-oriented modelling of the        modelling methods, the modelling-object classes being members of        a set of class libraries; writing, with the computing system,        quality-management classes using object-oriented modelling of        quality management, the quality-management classes being members        of the set of class libraries; scanning, with the computing        system, modelling-object classes in the set of class libraries        to determine modelling-object class definition information;        scanning, with the computing system, quality-management classes        in the set of class libraries to determine quality-management        class definition information; using, with the computing system,        the modelling-object class definition information and the        quality-management class definition information to produce        object manipulation functions that allow a quality management        system to access methods and attributes of modelling-object        classes to manipulate objects of the modelling-object classes;        and using, with the computing system, the modelling-object class        definition information and the quality-management class        definition information to produce access to the object        manipulation functions.    -   2E. The medium of embodiment 1E, wherein: executing quality        management comprises executing a process that integrates raw        data ingestion, manipulation, transformation, composition, and        storage for building artificial intelligence models.    -   3E. The medium of any one of embodiments 1E-2E, wherein: the        modeled quality management comprises management of extract,        transform, and load (ETL) phases of a machine learning model        designed in the OOM framework.    -   4E. The medium of any one of embodiments 1E-3E, wherein: the        modeled quality management comprises reporting of model        performance of a machine learning model designed in the OOM        framework.    -   5E. The medium of embodiment 4E, wherein: model performance is        measured by recall, precision, or F1 score.    -   6E. The medium of any one of embodiments 1E-5E, wherein: the        modeled quality management comprises data quality monitoring        (DQM).    -   7E. The medium of embodiment 6E, wherein: DQM comprises        monitoring data sources to detect a new or missing table or data        element, data element counts, data element null count and unique        counts, or datatype changes.    -   8E. The medium of any one of embodiments 1E-7E, wherein: the        modeled quality management comprises model quality monitoring        (MQM) of a machine learning model designed in the OOM framework.    -   9E. The medium of embodiment 8E, wherein: MQM comprises        measuring a model-based metric and causing model retraining        responsive to detecting more than a threshold amount of drift in        the model-based metric.    -   10E. The medium of any one of embodiments 1E-9E, wherein: the        modeled quality management comprises score quality monitoring        (SQM) of a machine learning model designed in the OOM framework.    -   11E. The medium of embodiment 10E, wherein: SQM comprises        performing a model hypothesis test.    -   12E. The medium of embodiment 10E, wherein: SQM comprises        computing a lift table or a decile table.    -   13E. The medium of any one of embodiments 1E-12E, wherein: the        modeled quality management comprises label quality monitoring        (LQM) of a machine learning model designed in the OOM framework.    -   14E. The medium of embodiment 13E, wherein: LQM comprises        determining which data sources among a plurality of data sources        are more leverageable or impactful on model performance than        other data sources among the plurality of data sources.    -   15E. The medium of any one of embodiments 1E-14E, wherein: the        modeled quality management comprises bias quality monitoring        (BQM) of a machine learning model designed in the OOM framework.    -   16E. The medium of embodiment 15E, wherein BQM comprises        detecting information bias, selection bias, or confounding by        the machine learning model designed in the OOM framework.    -   17E. The medium of any one of embodiments 1E-16E, wherein: the        modeled quality management comprises privacy quality monitoring        (PQM) of a machine learning model designed in the OOM framework.    -   18E. The medium of any one of embodiments 1E-17E, wherein: the        modeled quality management comprises data quality monitoring        (DQM) of a machine learning model designed in the        object-oriented modeling (OOM) framework; DQM comprises        monitoring data sources to detect a new or missing table or data        element, data element counts, data element null count and unique        counts, and datatype changes; the modeled quality management        comprises model quality monitoring (MQM) of the machine learning        model designed in the object-oriented modeling (OOM) framework;        MQM comprises measuring a model-based metric and causing model        retraining responsive to detecting more than a threshold amount        of drift in the model-based metric; the model-based metric is        indicative of an F1 score, accuracy, precision, mean error,        media error, distance measure, or recall; the modeled quality        management comprises score quality monitoring (SQM) of the        machine learning model designed in the object-oriented modeling        (OOM) framework; SQM comprises performing a model hypothesis        test and computing a lift table and a decile table based on        predicted probability of positive class membership, based on a        cumulative distribution function of positive cases; the model        hypothesis test comprises a Welch's t-test, Kolmogorov-Smirnov        test, or a Mann-Whitney U-test; the modeled quality management        comprises label quality monitoring (LQM) of the machine learning        model designed in the object-oriented modeling (OOM) framework;        LQM comprises determining which data sources among a plurality        of data sources are more leverageable or impactful on model        performance than other data sources among the plurality of data        sources; the modeled quality management comprises bias quality        monitoring (BQM) of the machine learning model designed in the        object-oriented modeling (OOM) framework; BQM comprises        detecting information bias, selection bias, and confounding by        the machine learning model designed in the object-oriented        modeling (OOM) framework; the modeled quality management        comprises privacy quality monitoring (PQM) of the machine        learning model designed in the OOM framework.    -   19E. The medium of any one of embodiments 1E-18E, wherein: the        modeled quality management comprises a process to determine data        source reliability.    -   20E. The medium of any one of embodiments 1E-19E, wherein: an        attribute of a quality-management object in one of the        quality-management classes comprise means for characterizing        quality with the attribute of the quality-management object.    -   21E. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system effectuate operations to measure the quality of        processing of data constructs in a pipeline using modelling        methods for implementation of machine learning design in an        object-oriented modeling (OOM) framework, the operations        comprising: forming, with one or more processors, a first        plurality of classes using object-oriented modelling of        modelling methods; forming, with one or more processors, a        second plurality of classes using object-oriented modelling of        quality measurement methods; accessing, with one or more        processors, a class library containing at least some of the        first plurality of classes to determine first class definition        information of a modeling method class among the first plurality        of classes; accessing, with one or more processors, the class        library or another class library containing at least some of the        second plurality of classes to determine second class definition        information of a quality measurement class among the second        plurality of classes; and using, with one or more processors,        the first and the second class definition information to produce        object manipulation functions that allow a computing system to        access methods and attributes of data construct classes to        manipulate data construct objects.    -   22E. The medium of embodiment 21E, the operations further        comprising: accessing the first and the second class definition        information to produce object manipulation functions that allow        the computing system to access methods and attributes of data        construct classes to manipulate data constructs; and accessing        second class definition information and third class definition        information of the data construct classes to produce object        manipulation functions that allow the computing system to access        methods and attributes of data construct classes to manipulate        said data constructs.    -   23E. A tangible, non-transitory, machine-readable medium storing        instructions that when executed by one or more processors in a        computing system effectuate operations to measure the quality of        processing of data constructs in a pipeline using modelling        methods for implementation of machine learning design in an        object-oriented modeling (OOM) framework, the operations        comprising: writing, with the computing system, a first        plurality of classes using object-oriented modelling of        modelling methods; writing, with the computing system, a second        plurality of classes using object-oriented modelling of quality        measurement methods; scanning, with the computing system, a        class library set containing a modelling method class among the        first plurality of classes to determine first class definition        information; scanning, with the computing system, a class        library set containing a quality management class among the        second plurality of classes to determine second class definition        information; and invoking, with the computing system, the first        class definition information of the quality management class and        the second class definition information of the modeling method        class to produce object manipulation functions that allow the        computing system to access the methods and attributes of data        classes to manipulate the modeling method class.    -   24E. The medium of embodiment 23E, wherein the object        manipulation functions comprise operations of: adding an        attribute to an object, deleting an attribute of an object,        updating an attribute to an object, reading an attribute of an        object, adding a reference to an object as an attribute,        changing an order of attributes, using a setter, or using a        getter.    -   25E. The medium of embodiment 23E, wherein the object        manipulation functions comprise operations of: formatting an        attribute, aggregating an attribute, calculating an attribute,        semantically altering attributes, aggregating an attribute,        contracting attribute, or expanding an attribute.    -   26E. The medium of embodiment 23E, wherein the object        manipulation functions comprise operations of: adding an        attribute to an object, deleting an attribute of an object,        updating an attribute to an object, reading an attribute of an        object, adding a reference to an object as an attribute,        changing an order of attributes, using a setter, using a getter;        formatting an attribute, aggregating an attribute, calculating        an attribute, semantically altering attributes, aggregating an        attribute, contracting attribute, and expanding an attribute.    -   27E. The medium of embodiment 23E, wherein object manipulation        is conditional.    -   28E. The medium of embodiment 23E, wherein the quality        measurement methods comprise: data quality, model quality, score        quality, bias quality, and label quality.    -   29E. The medium of embodiment 23E, wherein: entities captured in        data construct objects processed by the computing system include        consumers; communications to consumers by an enterprise;        communications to an enterprise by consumers; the events include        purchases by consumers from the enterprise; the events include        non-purchase interactions by consumers with the enterprise; and        the entity logs are obtained from a customer relationship        management system of the enterprise.    -   30E. The medium of embodiment 23E, wherein: the enterprise is a        credit card issuer and a trained predictive machine learning        models developed using the object-oriented modeling (OOM)        framework is configured to predict whether a consumer will        default; the enterprise is a lender and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will borrow; the        enterprise is an insurance company and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will file a claim; the        enterprise is an insurance company and the trained predictive        machine learning model developed using the OOM framework is        configured to predict whether a consumer will sign-up for        insurance; the enterprise is a vehicle seller and the trained        predictive machine learning model developed using the OOM        framework is configured to predict whether a consumer will        purchase a vehicle; the enterprise is a seller of goods and the        trained predictive machine learning model developed using the        OOM framework is configured to predict whether a consumer will        file a warranty claim, or the enterprise is a wireless operator        and the trained predictive machine learning model developed        using the OOM framework is configured to predict whether a        consumer upgrade their cellphone, or the enterprise is bank and        the trained predictive machine learning model developed using        the OOM framework is configured to predict the change in GDP.    -   1F. A method, comprising: the operations of any one of        embodiments 1A-30E.    -   2F. A system, comprising: the media of any one of embodiments        1A-30 coupled to one or more processors configured to execute        the instructions stored on the media.

What is claimed is:
 1. A tangible, non-transitory, machine-readablemedium storing instructions that when executed by one or more processorsin a computing system effectuate operations to measure performance ofprocessing of data constructs in a pipeline using modelling methods forimplementation of machine learning design in an object-oriented modeling(OOM) framework, the operations comprising: writing, with the computingsystem, a first plurality of classes using object-oriented modelling ofthe modelling methods; writing, with the computing system, a secondplurality of classes using object-oriented modelling of qualitymeasurement methods; scanning, with the computing system, a classlibrary containing modelling method classes to determine a first part ofclass definition information; scanning, with the computing system,another class library containing quality management classes to determinea second part of class definition information; and invoking, with thecomputing system, the class definition information to produce objectmanipulation functions that allow the computing system to access methodsand attributes of data classes to manipulate a modeling method class. 2.The medium of claim 1, wherein: quality monitoring (MQM), score qualitymonitoring (SQM), bias quality management (BQM), privacy qualitymanagement (PQM), or label quality monitoring (LQM).
 3. The medium ofclaim 1, wherein: the operations comprise object manipulation byallowing reading of attributes, usage of a given modeling method, auditof usage of a given modeling object, reporting attempts to use the givenmodeling object, or verifying proper licensing.
 4. The medium of claim1, wherein: the operations comprise object manipulation by allowingreading of attributes, usage of a given modeling method, audit of usageof a given modeling object, reporting attempts to use the given modelingobject, and verifying proper licensing; and the quality measurementmethods comprise data quality monitoring (DQM), model quality monitoring(MQM), score quality monitoring (SQM), bias quality management (BQM),privacy quality management (PQM), and label quality monitoring (LQM). 5.The medium claim 1, wherein: the operations further comprise processingdata construct objects based on entity logs; entities captured in thedata construct objects comprise: consumers, communications to consumersby an enterprise, communications to an enterprise by consumers, andevents that include purchases by consumers from the enterprise andnon-purchase interactions by consumers with the enterprise; and theentity logs are obtained from a customer relationship management systemof the enterprise.
 6. The medium claim 1, wherein: the enterprise is acredit card issuer and a trained predictive machine learning modeldeveloped using the object-oriented modeling (OOM) framework isconfigured to predict whether a consumer will default; the enterprise isa lender and the trained predictive machine learning model developedusing the OOM framework is configured to predict whether a consumer willborrow; the enterprise is an insurance company and the trainedpredictive machine learning model developed using the OOM framework isconfigured to predict whether a consumer will file a claim; theenterprise is an insurance company and the trained predictive machinelearning model developed using the OOM framework is configured topredict whether a consumer will sign-up for insurance; the enterprise isa vehicle seller and the trained predictive machine learning modeldeveloped using the OOM framework is configured to predict whether aconsumer will purchase a vehicle; the enterprise is a seller of goodsand the trained predictive machine learning model developed using theOOM framework is configured to predict whether a consumer will file awarranty claim; the enterprise is a wireless operator and the trainedpredictive machine learning model developed using the OOM framework isconfigured to predict whether a consumer upgrade their cellphone; or theenterprise is bank and the trained predictive machine learning modeldeveloped using the OOM framework is configured to predict the change inGDP.
 7. The medium of claim 1, wherein: the operations comprise stepsfor object-oriented orchestration.
 8. The medium of claim 1, wherein:the operations comprise steps for scaled propensity modeling.
 9. Themedium of claim 1, wherein: the operations comprise steps forObject-Oriented Modeling transformation from data to labeled data. 10.The medium of claim 1, wherein: the operations comprise steps forObject-Oriented Modeling composition of object-oriented pillars.
 11. Amethod comprising: writing, with a computing system, a first pluralityof classes using object-oriented modelling of modelling methods;writing, with the computing system, a second plurality of classes usingobject-oriented modelling of quality measurement methods; scanning, withthe computing system, a class library containing modelling methodclasses to determine a first part of class definition information;scanning, with the computing system, another class library containingquality management classes to determine a second part of classdefinition information; and invoking, with the computing system, theclass definition information to produce object manipulation functionsthat allow the computing system to access methods and attributes of dataclasses to manipulate a modeling method class.
 12. The method of claim11, wherein: quality monitoring (MQM), score quality monitoring (SQM),bias quality management (BQM), privacy quality management (PQM), orlabel quality monitoring (LQM).
 13. The method of claim 11, wherein: themethod comprises object manipulation by allowing reading of attributes,usage of a given modeling method, audit of usage of a given modelingobject, reporting attempts to use the given modeling object, orverifying proper licensing.
 14. The method of claim 11, wherein: themethod comprises object manipulation by allowing reading of attributes,usage of a given modeling method, audit of usage of a given modelingobject, reporting attempts to use the given modeling object, andverifying proper licensing; and the quality measurement methods comprisedata quality monitoring (DQM), model quality monitoring (MQM), scorequality monitoring (SQM), bias quality management (BQM), privacy qualitymanagement (PQM), and label quality monitoring (LQM).
 15. The methodclaim 11, wherein: the method further comprises processing dataconstruct objects based on entity logs; entities captured in the dataconstruct objects comprise: consumers, communications to consumers by anenterprise, communications to an enterprise by consumers, and eventsthat include purchases by consumers from the enterprise and non-purchaseinteractions by consumers with the enterprise; and the entity logs areobtained from a customer relationship management system of theenterprise.
 16. The method claim 11, wherein: the enterprise is a creditcard issuer and a trained predictive machine learning model developedusing the object-oriented modeling (OOM) framework is configured topredict whether a consumer will default; the enterprise is a lender andthe trained predictive machine learning model developed using the OOMframework is configured to predict whether a consumer will borrow; theenterprise is an insurance company and the trained predictive machinelearning model developed using the OOM framework is configured topredict whether a consumer will file a claim; the enterprise is aninsurance company and the trained predictive machine learning modeldeveloped using the OOM framework is configured to predict whether aconsumer will sign-up for insurance; the enterprise is a vehicle sellerand the trained predictive machine learning model developed using theOOM framework is configured to predict whether a consumer will purchasea vehicle; the enterprise is a seller of goods and the trainedpredictive machine learning model developed using the OOM framework isconfigured to predict whether a consumer will file a warranty claim; theenterprise is a wireless operator and the trained predictive machinelearning model developed using the OOM framework is configured topredict whether a consumer upgrade their cellphone; or the enterprise isbank and the trained predictive machine learning model developed usingthe OOM framework is configured to predict the change in GDP.
 17. Themethod of claim 11, comprising steps for object-oriented orchestration.18. The method of claim 11, comprising steps for scaled propensitymodeling.
 19. The method of claim 11, comprising steps forObject-Oriented Modeling transformation from data to labeled data. 20.The method of claim 11, comprising steps for Object-Oriented Modelingcomposition of object-oriented pillars.